THE EVALUATION OF SUCCESS IN 
NAVAL SHIP ACQUISITION 



Virgil Frank McCaleb 



DllHLEV KNOX LIBIU^ 
naval postgraduate 8CHCK)t 

V-.n-EREY. CAUFORNU 93940 



THE EVALUATION 



OF SUCCESS IN 
NAVAL SHIP ACQUISITION 
by 

VIRGIL FRANK MCCALEB 
B.S.^ University of New Mexico 

(1967) 

SUBMITTED IN PARTIAL FULFILD^ENT 
OF THE REQUIREMENTS FOR THE 
DEGREES OF 
OCEAN ENGINEER 
AND 

MASTER OF SCIENCE 
at the 

MASSACHUSETTS INSTITUTE OF 
TECHNOLOGY 
June, 1976 



*£ 



THE EVALUATION OF SUCCESS IN 



NAVAL SHIP ACQUISITION 

by 

VIRGIL FRANK MCCALEB 



Submitted to the Department of Ocean Engineering 
and the Alfred P. Sloan School of Management on 
May 7> 1976, in partial fulfillment of the 
requirements for the degrees of Ocean Engineer 
and Master of Science. 



ABSTRACT 

An overview of post World War II U.S, naval ship acqui- 
sition policies and results is presented. Hie major emphasis 
in the overview is on several new control systems and policies 
announced since 1970. 

A case for development of a comprehensive, life-cycle 
ship success evaluation system is based on the current domi- 
n?5nt emphasis on short-term goal achievement and the need for 
a centralized, complete historicai record and data bank on 
ship acquisitions. 

Success in naval v*^hip acquisitions is defined as how 
well actual results match a set of prioritized goals determ- 
ined from the requirements of the military need. Individual 
sets of criteria for performance, cost and schedule goals are 

developed. Two schemes for evaluating overall ship 

2 



acqiiisition success, a "manual" model and a computerized 
simulation model, are outlined. 

The development of a comprehensive life-cycle ship 
acquisition evaluation system is recommended. Further 
study and development of certain aspects of the proposed 
system are required. 
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INTRODUCTION 



The following excerpts are taken from an article in 
the March ky 1976, edition of the Boston Globe . 

” a shipbuilder (is) seeking payment of 

$11,708 owed by the Federal Government 

for a Navy (ship) the 300 foot (ship) 

cost the Navy $75,000, plus a $16,000 

overrun resulting from requested modifications 

and delays in subcontract work, the 

unpaid debt was a major reason why (the ship- 
yard) ultimately went bankrupt " 

Only the magnitude of the figures set this article apart from 
similar, almost daily expositions. Generally such paltry 
sums as above are not newsworthy. A $l60,000,000 overrun 
would be more like it. Actually the article refers to a 
claim filed by descendants of the shipbuilder over a disputed 
unpaid bill resulting from construction of a Union Navy gun- 
boat during the Civil War. As in todays similar reports, 
the ultimate "success" of the gunboat - or failure - is left 
unaddressed. 

In the conclusion of an exhaustive study by Dallen, 

Merkl and Zlatoper of the Army's ill-fated main battle tank 
(MBT-70) program, it is noted that various players in the 
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weapons acquisition process view "success" quite differ*' 

1 

entXy. The authors observed that the strength of "sponsor- 
ship" of a given weapons acquisition may Very well determine 
whether or not the system eventually becomes part of the 
nation's arsenal, regardless of the actual need or degree 
of problems encountered in the development and acquisition. 
The study authors, therefore, supported the "adoption of a 

set of 'rational' criteria by which to judge system suc- 
2 

____ II 



THESIS OBJECTIVES 

Improving the results of defense systems acquisition 
depends on learning from past mistakes and past successes. 
This requires consistent evaluation of project results with 
as much objectivity as possible. 

The purpose of this thesis is to investigate the possi- 
bility of such a system of evaluation of the degree of suc- 
cess in naval ship acquisition. The objectives then are to 
determine for naval ship acquisitions a criterion for success 
and investigate the feasibility and alternatives for such a 
system. 



T 

Footnotes and references are listed at the ends of the 
chapters. 
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THE COMPLEXITY OF THE HARDWARE 



Articles on naval ship acquisition usually begin with 
acknowledgment that major defense weapons systems are the 
most technically complex of any in existence. Perhaps the 
most complex of weapons systems are naval ships. 

Ships are the largest mobile objects on Earth and naval 
vessels represent an integration of a myriad of major and 
minor interlocking systems of which many are extremely com- 
plex in their own right. A nuclear powered aircraft carrier 
is over 1100 feet long, displaces over 90,000 tons, and is 
propelled at speeds in excess of 30 knots by over 200,000 
shaft horsepower. It is an airport with a capacity of 100 
or more Jet aircraft. Additionally, it is a self contained 
city with a population of 5,000 to 6,000 people. Its multi- 
reactor nuclear plant, various electronic systems and air- 
craft launching and recovery systems take years to design 
and prove before they are ready for installation. 

THE COMPLEXITY OF THE PROBLEM 

The complexity of the hardware in question is perhaps 
only matched by the complexity of the bureaucracy and pro- 
cess concerned with the acquisition of the hardware. In 
World War II, the entire Navy Department in Washington, 

D.C, charged with directing a war effort involving the 



peak of the war about 50,000 ships, consisted of about 200 
people. Today there are in excess of 20,000 in the Washings 
ton, D.C., metropolitan area employed by the Navy. The ac- 
tive fleet today consists of less than 500 ships. 

It commonly takes ten to twelve years to conceive, 
develop, design and construct a new U.S. Navy ship class. 

The problems of long range fiscal and engineering forecast- 
ing and planning in an unstable political and economic envir 
onment; and in the face of rapid technological advances, can 
be ^'mind boggling” as one Ship Acquisition Project Manager 
observed to the author. 



THESIS OVERVIEW 

The first chapter of the thesis is concerned with the 
three rather distinct strategies employed in defense systems 
acquisition since World War II. Though certainly not a com- 
plete treatment, the reader should gain from Chapter I con- 
siderable background information about the naval ship (and 
other weapon systems) acquisition process and structure. 

The major emphasis is on several new systems and policies 
which have been formulated during the past five or six years 
The reader interested in a more complete analysis of the 
defense weapons acquisition process, particularly during the 

1960 ’s, is referred to the recent and excellent book 

3 

Arming America by J. Ronald Fox. 
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Chapter II is concerned with the case for a comprehen- 
sive, consistent evaluation system for naval ships. The 
problem of acquisition projects outliving acquisition poli- 
cies and strategies is discussed. Then, the existing over- 
emphasis on "short-term” goal achievement, and the reasons 
for that over-emphasis, at the expense of longer term, stra- 
tegic goals, is investigated. 

It is concluded in Chapter II that a comprehensive eval- 
uation system, could help shift the balance of goal emphasis 
as well as fill the need for a centralized, complete histori- 
cal record and data bsoik for ship acquisitions. A position is 
taken against use of the system for personnel performance 
eva luation. 

Chapter III begins with noting that relevancy, objectiv- 
ity and feasibility are desirable traits of a success evalua- 
tion system, but that total objectivity (requiring relevant, 
strictly quantitative criteria) is unobtainable in a feasible 
evaluation system for such a complex system. 

The specification of goals and establishment of their 
relative priorities from the need requirements are noted as 
being prerequisites to the evaluation system. A definition 
of success is also required and a definition preference for 
naval ship acquisitions is offered. 

The remainder of Chapter III is devoted to developing 
sets of criteria for the three major goal areas; performance. 
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cost and schedule. Finally, two different approaches to 
combining the individual criteria so that an overall assess- 
ment of success is possible are discussed. 

A manual evaluation model is outlined and the reasons 
for adopting such a "low resource requirement" approach are 
offered. The scheme is similar to those utilized by acquisi- 
tion projects for evaluating contractor bid proposals, 

A framework for the development of a much more sophisti- 
cated, computerized simulation model is presented as an 
alternative to the manual model. This approach requires 
more resources to develop and use, but offers the possibility 
of a more accurate and objective model as more of the complex 
interdependencies of an acquisition^ can be accounted for 
and probabilistic criteria can be fully utilized. 

The final conclusions and recommendations include 

a) the success evaluation system outlined in thfe 
thesis is imperfect but preferable to the current situation 
(i.e. no long term structured system at all), 

b) more work needs to be done in constructing evalua- 
tion criteria and in implementing existing and planned ’systems 
upon which the proposed evaluation system is built, 

and, 

c) the further specification and trial implementation 
of the proposed system of success evaluation is recommended. 
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1. J. A. Dallen, Jr., L. C. Merkl, and R. J, Zlatoper, Mana« 
ging the Weapon Systems Acquisition Process - A Case Study 
of the Main Battle Tank - 70, unpublished Masters of Science 
Thesis (Oambrldge, Massachusetts, Sloan School of Management, 
M.I.T. , 1975). 

2. Ibid. 

3. J. Ronald Pox, Arming America : How the U. S. Buys Weapons , 
(Boston, Division of Reasearch, Harvard Business School, 
1974). 
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CHAPTER I 



AN OVERVIEW OF POST WWII U.S. NAVAL SHIP ACQUISITION 
INTRODUCTION 



A brief overview of the structure and process of naval 
ship acquisition in the United States since World War II is 
presented in this chapter. Three rather distinct periods are 
indet if table - the "Conventional" Period (until the early 
1960*5), the "Concept Pormulation/Contract Definition" or 
"Total Package Procurement" Period (also called the "McNamara" 
Period after the then Secretary of Defense Robert S. McNamara) 
which began in the early 1960*s and ended about 1969> and the 
"Current" Period. The major policies and characteristics of 
the three periods differ considerably, but it is not always 
possible to categorize a particular ship acquisition project 
as being a result of the policies of any one particular peri- 
od. For example, the nuclear powered aircraft carrier pro- 
ject (the NIMITZ class of ships) was conceived during the 
Conventional Period, continued through the Concept Pormula- 
tion/Contract Definition Period, and remains an ongoing pro- 
ject. The project has characteristics of all three policy 
periods. The ships were designed primarily by the Navy ("in 
house") with the aid of a design agent, typical of the Con- 
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ventional and Current Periods. The three (or four) ships of 
the class are being constructed under a multi-ship, multi- 
year contract, a significant aspect of the Concept Formula- 
tion/Contract Definition Period which has been continued. 

Thus as the different periods are described, it should 
be remembered that ship projects are long (often ten to 
twelve years) and often transcend major acquisition policy 
shifts. Additionally, like any large bureaucracy, new poli- 
cies and strategies from top management (the Secretary of 
Defense) often do not taice effect at the working level (the 
projects) for two or three years, if at mil. 

THE BASIC PROCESS 

Although policies and the organizational structure 
for designing and acquiring ships for the U.S. Navy have 
changed over the years, the basic process remains much the 
same* Also, though differing in details and nomenclature, 
the acquisition of ships is similar to the acquisition of 
other major defense systems. A need is identified; a re- 
quirement based on that need is established; a weapon sys- 
tem is selected, developed, designed and constructed to fill 
the requirement. Sometimes technological breakthroughs 
motivate a new acquisition (termed "technology push") but 
attempts are made to ensure that a legitimate need exists 
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prior to development and construction of the system. 

The Navy and the other services are charged with "iden- 
tifying needs and defining, developing and producing systems 

1 

to satisfy those needs." Establishing overall acquisition 

policy, passing on the validity of needs, and monitoring the 

performance of the services in carrying out the policy is 

2 

the purview of the Office of the Secretary of Defense. 

National defense policies and objectives are provided by 
the Secretary of Defense and translated by the Joint Chiefs 
of Staff into military policies and objectives. Planning and 
programming by the services is keyed to these strategic ob- 
jectives. Evaluation of the Joint Chiefs of Staff guidance 
can lead to research and development objectives formulation 
by the services to satisfy deficiencies in their capabili- 
ties to perform their respective assigned missions and . 

3 

roles. The Program Objectives Memorandum is the budget for 
this effort. Later, Program Objectives Memoranda will con- 
tain the weapons systems which emerge from the research and 
4 

development efforts. 

The Program Objectives Memoranda are part of the Depart- 
ment of Defense Planning- Programming- Budge ting System (PPBS). 
Funding for weapons systems is obtained through the PPBS. 
However, a series of approvals by intra-service councils 

^ 

Notes are listed at the end of each chapter. 
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and the top level Defense Systems Acquisition Review Council 
is currently also required before a new weapons system is 
constructed. The role of the Defense System Acquisition 
Review Council will be discussed later in the chapter. 

THE CONVENTIONAL PERIOD 

At the end of World War II the U.S. Navy had 50,000 
ships and there were 50 private and public shipyards work- 
ing at or near capacity. Much of the fleet was "mothballed” 
(retired from active service) at the end of the war and pro- 
duction virtually ceased at the shipyards. During the Korean 
conflict most of the required ships were reactivated from 
the "mothball fleet". It was not until 1952 that the first 

significant new shipbuilding since World War II - 31 major 
5 

ships - was directed. 

The major policies and practices were characterized 
by an iterative design process accomplished by the Navy 
"in house” or by an independent design agent up to amd includ- 
ing a complete construction bid package, little docximenta- 
tion, major emphasis on ship performance, the splitting of 
the production contracts between two or more shipbuilders, 
and little involvement by the Office of the Secretary of 
Defense. The acquisition process was basically decentral- 
ized to the service level. 
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Initially, the entire design and procuriCinent effort 
would be coordinated by a few individuals. They relied 
almost completely on the functional organizations to perform 
the necessary design and acquisition work required. Differ- 
ent self-contained organizations would be responsible for 
various systems on the ship. The ”Gun Club” (Bureau of 
Ordnance) was responsible for weapons for example. Later, 
starting with the Polaris and other ballistic missile pro- 
greons, the trend was towards project manager type organiza- 
tions with varying degrees of self containment. 

The reasons for spreading the production contracts 
between several shipbuilders are not known for certain but 
probably included the desires for rapid delivery of the ship 
class and preservation of the production base of the ship- 
yards. Alternatively, the primary reason may have simply 

been regional political/economic pressures. 

6 

As depicted in Figure 1-1 , the ’’Conventional" approach 

typically consisted of "in-house" talent formulating a ship 

concept. Ihis activity Included cost and feasibility studies 

and advance research and development as necessary. Assuming 

budgetary approval, increasingly refined and detailed design 

stages, termed "Preliminary Design" and ’bontract Design" 

followed. The approach was basically evolutionary and util- 

7 

ized minimal systems analysis techniques. 
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SHIPBUILDERS 



The resulting product was a complete bid package, in- 
cluding complete contract plans and specification. The 
basic bid package could result in any number of procurement 
contracts. Lead ships (the first of a class) were often 
constructed in Navy shipyards. The amphibious ships LPD 
7 through 15 were built under four separate contracts by 
two shipbuilders. Exclusive of the costs of changes to the 
contract, these ships were delivered to the Navy at an 
average 125 ^ of the initial contract grice (fixed price) and 
an average 27 months behind schedule. Escalation due to 
inflation and claims against the government by the contrac- 
tors accounted for most of the average 25^ cost overruns. 

The above results became increasingly typical. Low or 
negative profit performance by the contractors is thought to 
have precipitated many of the claims. The basis of conten- 
tion in the claims was usually a dispute over interpretation 
of the often complex and detailed contract specifications. 
Additionally, production facilities were becoming antiquated 
and uncompetitive in the world market. Support of the ships 
was costly and often inadequate due to a lack of standardi- 
zation between ships of a given class and between classes 
of ships. 
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THE CONCEPT FORMULATION/ CONTRACT DEFINITION 
TOTAL PACKAGE PROCUREMENT PERIOD 



A radically different approach to weapons design and 

acquisition was formulated in the early 196 o*s by the 

Office of the Secretary of Defense under Robert S* McNamara. 

The new approach centralized major decision making authority 

9 

to the Office of the Secretary of Defense. Objectives were 

a) optimization of cost effectiveness by using systems 
analysis techniques, 

b) reduction or elimination of contractor claims 
against the government by using contractor prepared perform- 
ance oriented specifications vice government imposed detailed 
specifications , 

c) reduction of cost overruns by transferring finan- 
cial risk to the contractors for the design and acquisition 
phases through the use of fixed price contracts ^ , 

d) significant capitalization increases in shipbuild- 
ing facilities by using multi-ship, multi-year contract 
awards to a single shipbuilder that were expected to pro- 
vide long term financial security enabling large scale capi- 
talization ajid force expansion of facilities due to delivery 
schedule demands, 

e) reduce \mique systems and subsystems proliferation 
resulting from split production contracts. 
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f) introduce producability and innovation into the 
designs by having the production contractor design the 
system, 

g) lower acquisition costs by talcing advantage of the 
"Learning Curve” effect through single producer serial pro- 
duction (The "Learning Curve” had long been thought not to 
apply to shipbuilding but several studies conducted during 
the 1950 *s and 1960’s indicated otherwise.), 

and, 

h) arrive at more accurate total cost estimates and 
reduce poor ship support by making the contractor responsi- 
ble for all on board systems, crew training, initial repair 
parts, and support facilities. (This policy is known as 
"Total Package Procurement”. Strictly speaking neither of 
the Navy acquisitions being acquired under Concept Pormula- 
tion/Contract Definition policies are "Total Package Pro- 
curement" as there is government furnished equipment involv- 
ed, ) 

A project manager type organization was directed for 
10 

all major programs . As outlined in Figure I-l, the ser- 
vices would still conduct advanced research and development 
and identify the desired performance characteristics of the 
new weapon system during the Concept Formulation stage. 
Assuming approval by the Secretary of Defense, a Contract 
Definition period would follow. A "Request for Proposal" 
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was prepared by the service and issued to selected shipbuil- 
ders (when more than one were thought capable) to prepare' 
design analyses based on the specified performance charac- 
teristics. The Request for Proposal contained both manda- 
tory and desirable performance specifications and were sup- 
posed to "encourage alternatives and stimulate initiative 

11 

and creativity by the contractors". 

After evaluation of the proposals by the service, nor- 
mally two or more contractors were given fixed price con- 
tracts to develop a complete shipbuilding proposal. Required 
in these proposals were contract plans and specifications, 

detailed construction plans, management plans and a complete 

12 

analysis of the Life Cycle Costs. Life Cycle Costs meant 

the total costs of acquisition and ownership - development, 

13 

production, deployment, operation and maintenance. 

Total system tradeoffs were encouraged "to obtain, 
within the mission and performance envelopes, an optimiuft 
balance between total cost, schedule, and operational effect- 
iveness for the system". Operational effectiveness was defin- 
ed to include "pure" performance, reliability, maintainabil- 
ity and all other factors which affected the effectiveness of 
the new system. The system was defined to Include the re- 
quired facilities, personnel, data, training equipment as 

14 

well as the the hardware Itself. 



No longer than six months was allowed for the Contract 
Definition phase. This was followed by a Source Selection 
process during which a detailed analysis of the proposals 
was conducted by the procuring service. Negotiation was con- 
ducted with one or more of the contractors and transfusion 
of parts of one proposal into another was permitted. At the 
end of the evaluation period a recommendation was sent to 
the Office of the Secretary of Defense to award a multi- 
year, multi-ship procurement contract to the contractor with 
the best proposal, to conduct further Contract Definition, 
or to defer or abandon the effort (unlikely). The single 
contract awarded was fixed price, with or without incentive 
clauses. 

The Navy conducted three Concept Formulation/ Contract 
Definition ship competitions. The Fast Deployment Logistics 
Ship (PDL class) was not funded by Congress. The Amphibious 
Helicopter Assault ships (LHA class) and the SPRUANCE class 
destroyers (DD-963 class) are currently under construction. 
Litton Industries won all three competitions. 

CONCEPT FORMULATION/CONTRACT DEFINITION RESULTS 

"Frankly gentlemen, in defense procurement, we 
have a real mess on our heuids." 

The Honorable David Packard 
Deputy Secretary of Defense, 1970 

It is too early to evaluate the final results of the 

LHA and DD-963 shj.p acquisitiona The USS SPRUANCE (DD-963) 

24 



is the first ship delivered under either contract and was 
accepted by the Navy in 1975 • However, the acquisitions have 
been beset by much the same problems that have characterized 
defense weapons systems procurement the last two decades - 
acquisition cost and schedule overruns of large proportions. 
This is particularly true of the first of the contracts, 
the LHA class. The first LHA is yet to be delivered and is 
several years behind the original schedule. A large claim 
against the government by Litton over the contract is pend- 
ing and the contract has already been renegotiated (upwards 
in price and longer delivery times). 

By the late 196o*s, cost and schedule overruns and 
performance shortfalls of new major weapons systems were 
daily newspaper fare. In 1971 the Assistant Secretary of 

Defense for Financial Management (Comptroller) conducted a 

16 

survey of 35 major development and production programs. 

Only two of the programs were found to be on, or ahead of, 
schedule. That same year the General Accounting Office 
made a survey of 6l weapon systems and found that cost esti- 
mates for them had increased $33*4 billion over the initial 

17 

estimates. Contractor costs soared and profits plummeted. 
Huge cost and schedule problems in the C-5A aircraft were 

overshadowed in 1971 when cracks developed in the engine 

18 

mountings. The term "contractor bailout" became a house- 
hold word as one producer after another threatened to cease 
production unless relief from the fixed price contracts was 
provided. ?5 



5y 1970 a number of studies had found serious flaws 
in the management of the weapons acquisition process. 

The President’s Blue Ribbon Panel, The National Security 
Industrial Association, the Defense Science Board Task 
Force on Research and Development Management, the Aero- 
space Industries Association, and the General Accounting 
Office conducted studies which all sounded very much the 
same theme. A March 1970 Logistics Management Institute 

19 

report summarized the most common criticisms as follows; 

” First is the observation that the weapons 
acquisition process apparently is out of 
control. Initial time and cost estimates - 
and even updated estimates - cannot be depen- 
ded upon. Mandatory engineering changes arise 
continually throughout the process. Manage- 
ment information and control systems do not 
identify impending problems in time for pre- 
ventive action to be taken. 

Second is the claim that bargaining positions 
are unbalanced; first one side, the the other 
has the advantage. The theory of counter- 
vailing pressures acting to produce fair and 
realistic contract terms does not hold. With 
emphasis on economies of scale and series pro- 
duction there are only a small number of 
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weapon systems competitions each year and 
prospective contractors believe that their 
very existence may be jeopardized by failure 
to win. Hence the Department of Defense (DOD) 
is in the dominant position and can compel 
an unreasonable bargain. Following award 
of the contract, the DOD, committed to the 
timely success of the program, is in the 
weaker position as the sole source contrac- 
tor negotiates for contract changes, prod- 
uct acceptance, and follow-on business. 

Third is that incentives both for effi*- 
cient operation and for candor about expec- 
tations are lacking. Heavy reliance on 
historical costs in pricing, lack of ade- 
quate consideration of capital required 
in negotiating profit rates, and the high 
risk of low future utilization of contrac- 
tor owned facilities impede investment and 
modernization of plant. The hazard to pro- 
gram survival, of high cost, long duration, 
or looming technical difficulties, as each 
program competes with others in and out of 
the DOD, motivates extreme: optimism by 
DOD and contractor personnel alike. 
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Fourth are allegations of confusion, con- 



nivance, and deception by the DOD-con- 
tractor combination. Close cooperation 
and common interest are held in contrast 
to the arm*s length relationship prefer- 
red by much of regulation and policy. 

Policy notwithstanding, the military 
departments receive advice and assistance 
from prospective contractors in prepara- 
tion of Requests for Proposals. Contrac- 
tors receive aid from government person- 
nel in performance of contracts. Contracts 
fail as instruments of control.” 

On May 28, 1970, Deputy Secretary of Defense David 
Packard issued a memorandum which stated that the Total 
Package Procurement approach to developing and acquiring 

major weapon systems was unsatisfactory and that a new 

20 

policy would soon be established. The Navy was still 
years away from delivery of its first ship procured under 
the cancelled policy. Whether or not the long term objec- 
tives of the Concept Formulation/ Contract Definition/Total 
Package Procurement Period will be achieved for the Navy 
ships remains to be seen. It is a fact that the sole source 
multi-year contracts resulted in the construction by Litton 
of a new shipyard at Pascagoula, Mississippi. Interviews 



28 



conducted by the author with project personnel brought out 
the beliefs of several that the policies were never given a 
fair chance. This resulted from the failures of the P-111 
and C-5A aircraft procurements in particular and the problems 
that Litton was experiencing in developing an adequate design 
and production force and in making the new facility operai*^ 
tlonally efficient. 



THE CURRENT PERIOD 

The major policies and trends of new ship acquisition 
since the demise of Concept Formulation/ Contract Definition 
include 

a) emphasis on constrained design ( "Design- to-Cost'*), 

b) emphasis on proven hardware ("Ply-before-you-Buy" ), 

c) required review and approval to proceed by the 
Defense Systems Acquisition Review Council at key milestones, 

d) a prohibition against Total Package Procurement 
(linkage of system development and production under a single 
contract), 

e) Improvement of the quality and validity of cost 
estimates, 

f) flexibility in contract type and liberalization 
of contract escalation (due to Inflation) clauses, 

g) use of contractor aided "in-house" ship design. 



and 



h) tailoring of acquisition appr^>aches to each pro-- 

ject. 



"DESIGN-TO-COST” 

21 

The cancellation of the key top level policy directive 
for the Concept Formula tion/Con tract Definition period in 
1970 left a guidance void that was not formally filled until 
the issuance of Department of Defense Directive 5000>.l, 
"Acquisition of Major Defense Systems", on July 13 > 1971. 

It was during this same period that then Chief of Naval 
Operations Zumwalt directed the rapid development of a large 
class of austere,' relatively Inexpensive Guided Missile Fri- 
gates (FFG class, formerly the Patrol Frigate or PF class) 

22 

to bolster the size of the rapidly diminishing fleet. 

After a year of feasibility studies. Admiral Zumwalt 
directed that the design would not violate constraints which 
were set on the average "follow" (after the first) ship 

23 

acquisition cost and fully loaded ship displacement. He 

later added a third constraint on the maximum number of 

24 

accomodations on the ship. Performance capability above 
the minimum specified was to be traded off to stay within 
the constraints. This method of ship design, commonly termed 
"Design- to-Cost", was revolutionary to the Navy, but is com- 
mon in industry for new product development. 
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A major program consideration of current acquisition 
policy is that "discrete cost elements (e.g. unit produc- 
tion cost^ operating and support cost) shall be translated 

25 

into "design to" requirements". In October 1973 the major 

services’ material commands Jointly issued a "Joint Design- 
26 

to-Cost Guide". This directive requires that "Design-to- 
Cost be used for major systems except that 

a) "Major weapons systems which for reasons of 
national security necessarily push the state-of-the-art 
should stress performance. In such cases, unit production 
cost goals will be applied and tracked but will be subordi- 
nate to performance goals.” 

and , 

b) "Major weapons systems where the assessed need 

is sufficiently urgent should stress the schedule. In such 
cases, unit production cost goals will be applied and tracked 
but will be subordinate to schedule goals." 

Implementation of the above exceptions requires Secretary of 
Defense approval. The TRIDENT strategic ballistic missile 
submarine class is such an example. 

Historically, performance requirements for new ships 
had been dictated by the Chief of Naval Operations to the 

28 

material command in brief "single sheet characteristics". 
These were used by the material command "to develop prelim- 
inary designs and cost eftimates leading to more detailed 
characteristics statements, and ultimately to procurement 
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Costs were considered but were usually 



specifications . 
secondary to maximizing performance. As "Design- to-Cost” 
elevated the importance of acquisition cost to the same level 
as performance in the design process, a new iterative perfor- 
mance-cost tradeoff dialogue between the customer (Chief of 
Naval Operations) and producer (Chief of Naval Material) 
organizations was required. "Top Level Requirements and 
Top Level Specifications for the development of naval ships". 
Chief of Naval Operations Instruction 9010. 300, was issued 
early in 1974. It details a procedure which provide® for a 
working group to develop the performance parameters for a 
baseline ship which will meet the established mission re- 
quirements. The group also specifies allowable variations 

in performance parameters and alternative system selections 
30 

for the ship class. 

After a period of Feasibility Studies during which the 
impact of the alternative performance parameters and systems 
selections are evaluated, the Chief of Naval Operations 
organization starts preparation of a Draft Top Level Require- 
ments document. This document is revised as the Naval Mater- 
ial Command (parent command of the Naval Sea Systems Command 
for whom acquisition projects directly work) develops a con- 
ceptual design for the ship class and provides cost and 
design information to the Chief of Naval Operations. 

Assuming approval by the Chief of Naval Operations to proceed 
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with a selected design, a ’’Conceptual Baseline” and a "Cost 
Goal” for the average follow ship acquisition costs are pre- 
sented to the Defense Systems Acquisition Review Council 
which is composed of high level officials in the Office of 
the Secretary of Defense. If council and Secretary of 
Defense approval is given to proceed into ’’Preliminary 
Design”, a Draft Top Level Specifications document is started 
by the Naval Material Command (actually the acquisition 
project). This companion document to the Top Level Require- 
ments ’’translates the Top Level Requirements into a physical 
31 

ship description”. The contents of the two documents are 

discussed in Chapter III. The process phasing is presented 

32 

graphically in Figure 1-2. 

The Naval Sea Systems Command recently issued a ’’Design- 
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to-Cost Guide for Ship Acquisition . Ihis directive spe- 
cifies that the procedure of ’’Desgin- to-Cost” will be utili- 
zed for all ship acquisitions for which a Top Level Require- 
ments document is prepared. It exempts, as do most ship 
acquisition directive^ nuclear propulsion plants and nuclear 
support facilities under the cognizance of Admiral Hyman G. 
Rickover, Director of Naval Reactors. The Chief of Naval 
Operations has stated that a Top Level Requirements and Top 

Level Specifications will be developed for ’’all new ship de- 

34 

signs except fleet ballistic missile submarines.” 
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FLY- BEFORE- YOU- BUY 



It had been intended that before permission to proceed 
into Contract Definition would be granted, ’’primarily engine- 
ering rather than experimental effort” would be required and 

34 

the technology needed would be ’’well in hand”. However 

the large performance shortfalls, schedule delays and cost 

increases referred to in the previous section were at least 

partially a result of overly optimistic estimates of ultimate 

system capabilities and the time required to design and 
36 

perfect them. There had been a great deal of reliance on 
’’paper studies” rather than on actual performance demonstra- 
tions. Thus, a major program consideration of the Current 
37 

Period is that 

’’Programs shall be structured and resources 
allocated to ensure that the demonstration 
of actual achievement of program objectives 
is the pacing function. ” 

A supporting Department of Defense directive was issued in 

January of 1973 to establish test and evaluation policy for 

38 

the acquisition of defense systems. This policy emphasizes 
a) the need to conduct test and evaluation as early 
as possible and to continue it as long as necessary to elim- 
inate risks, 
and. 
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b) acquisition schedules will be based on the comple- 
tion of test and evaluation milestones prior to key decision 
points (Defense Systems Acquisition Review Council approval 
points. See Figure 1-2.) 

The key element which has grown from recognition of the 
need for increased test and evaluation during the acquisition 
process has been prototyping. This is commonly known as the 
"Fly-before-you-Buy” policy and is currently being used in 
the acquisition of new aircraft and in the Navy’s surface 
effect ships and NATO patrol hydrofoil programs. It is not 
feasible, however, to build and evaluate prototypes prior to 
beginning follow ship design and production for large ships. 
The time required, small numbers involved (one or two in some 
cases) and threat of obsolescence dictate a modified “Fly- 
before -you-Buy*’ approach. 

The Guided Missile Frigate Program developed a plan 
which provided for 

a) early construction of Land Based Test Sites for 
complete propulsion and combat systems 

and, 

b) a delay of two years between construction contract 

39 

awards for the lead and follow ships. 

This plan permits testing of the two major "high risk” 
subsystems prior to installation on the lead (first) ship and 
time to incorporate changes resulting from the entire test 
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and evaluation program into the design of the follow ships. 
The Land Based Test Sites will also be used for crew trainv 
ing. 



THE DEFENSE SYSTEMS ACQUISITION REVIEW COUNCIL 

T!tie Defense Systems Acquisition Review Council (DSARC) 
was established in the Office of the Secretary of Defense 

40 

by Deputy Secretary of Defense David Packard in May, I969. 

The purpose of the council reviews is to -^thplWl^ 

the status of major defense systems acquisition projects at 

three critical milestones which generally occur for ship 

acquisitions as indicated in Figure 1-2. 

Formal documentation for the DSARC reviews and decisions 

is provided by the project prepared Decision Coordinating 

Paper (formerly Development Concept Paper). It is a s\unmary 

document of not more than twenty pages that records the 

primary information on a program including the thresholds, 

issues and risks, need, the alternatives, the reviews, 

rationale for the decision, and affordability. When signed 

by the Secretary of Defense, it is the authority for the 

service to proceed with the program (or whatever action the 

Secretary of Defense directs). His decision sets the limits 

of authority (thresholds) within which the project is obliga- 
4l 

ted to stay. 
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A long series of intra-service briefings and reviews 
is generally required of an acquisition project prior to a 
DSARC presentation. 



COST ESTIMATING 

By 1970, the credibility of the Department of Defense 
concerning system acquisition cost estimates was extremely 
low. When the new acquisition policy directive was issued 
by Mr. Packard in 1971, it directed that "Traceability of 

estimates and costing factors be maintained". In 

December, 1971, Mr. Packard issued a memorandum requirlftg 
incorporation of independent (from program proponents) para- 
metric cost analysis in Defense Systems Acquisition Review 

43 

Council presentations. 

Parametric cost estimates are "based on quantified 

relationships between the cost and physical performance 

characteristics of past systems for initial program decls- 
44 

ions". Engineering cost estimates (normally used by acqui- 
sition projects) are based on detailed design information. 
Cost Information Reports (now Contractor Cost Data Reports) 
submission requirements have been written into production 
contracts in order to build a data base from which to develop 

45 

parametric cost estimating relationships. 

Pox discusses a third estimating alternative for use in 

46 

preparing contract negotiation objectives. This technique, 
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known as "should cost" analysis had its beginnings in the 
civilian economy and was developed in the early i960 *6 for 
the Department of Defense; under the direction of then Defense 
Comptroller Robert N. Anthony. Fox describes the process as 
follows 

"This technique begins with an in depth analysis 
of a contractor’s management, cost estimating, and 
production practices to identify and measure the 
effects of poor performance. Prom a baseline devel- 
oped by eliminating costs resulting from inefficien- 
cies, projections are made for the current procure- 
ment. The objective is to foster improved industrial 
practices by setting realistic contract prices." 

Under the approach, a team of experts in engineering, 
pricing, auditing, procurement and management spend several 
weeks at the contractor's facility. They review in detail 
the company's operations and procedures. Standard industrial 
engineering techniques are used to uncover inefficiencies. 

Fox cites several examples of should-cost analyses 
successes in reducing costs. He also notes that it is dif- 
ficult to achieve high quality, factual investigations and 
that there had been considerable resistance from within 
contractor organizations and from within the Department of 
Defense . 

A Cost Analysis Improvement Group (CAIG) was established 
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in January 1972 in the Office of the Secretary of Defense 

"to review the estimates presented and to develop uniform 

47 

criteria to be used By the time a permanent char* 

ter for the CAIG was issued in 1973> its responsibilities 

had grown to include, interalia, identification of where 

efforts are needed to improve the cost estimating capabili- 
48 

ties of the services. Improvement of the validity and 
tracking of cost estimates has recently been addressed by 
the Secretary of the Navy, Chief of Naval Operations, etnd 

49 

Chief of Naval Material directives. None of the directives 

refer to the "should-cost" technique of cost estimating. 

A discussion of the provisions of the directives is contained 

in Chapter III. Figure 1-3 shows the responsibilities for 

ship cost estimating and the cost estimating organizational 

50 

relationships within the Navy. 

CONTRACTOR COST AND SCHEDULE CONTROL 

As more and more contractors failed to perform under 
the Concept Formulation/ Contract Definition Period fixed 
price contracts, the need for increased government engage- 
ment with the contractors was realized. The fixed price 
contracts and lack of government Involvement in the design 
supposedly transferred any financial risks from the govern- 
ment to the contractor and thus the role of the acquisition 
projects was basically that of monitoring, with little con- 
trol leverage. 
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(SHIP ACQUISITION CHAIN OP COMMAND 

PROJECTS )- Englnee ring 

gQst estimates | PIGURE 13 WORKING RELATIONSHIP 



The current top level acquisition policy directive spe<- 

cifles that ’’Contract type shall be consistent with all 

program characteristics including risk.” Also stipulated is 

that "Cost type prime and subcontracts are preferred where 

51 

substantial development effort is involved.” The use of 

cost type contracts opens up the possibility of increased 

government engagement with the contractor. 

Apart from the lack of governmental control leverage 

resulting from fixed price contracts, attempts at effective 

contractor cost and schedule control by the projects has 

52 

historically been hampered by 

a) a reluctance of the contractors to share what it 
considers to be proprietary information, 

b) the preoccupation of project managers with the 
annual funding approval process and the continuity of funds 
flow (funds control) as opposed to cost control, 

c) the proliferation of various information and cost 
control systems' imposed on contractors by the different 
services and projects (validity of the information was often 
lost in the translation from the contractor's system to the 
government imposed system(s)), 

d) the lack of inclusion in the reporting systems of 
the budgeted value of work performed, 

e) improper allocation of contractor costs between 
overhead (indirect) and direct costs. 
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f) inability of the project personnel to evaluate 
the detailed information they require of the contractor, aind 
lack of correspondence between reported data and the contrac- 
tor *s own data, 

g) retroactive changing of financial plans to conform 
to work performed to date (the "rubber baseline") 

and, 

h) contractor use of nonintegrated work breakdown 
structures and nonintegrated charts of cost accounts (sum 
of budget dollars for work at one level may exceed budget 
at next higher level). 

In a survey conducted during the 1960*s, Pox found that 

most program managers were satisfied if their funds control 

reports indicated that funds were being expended at the plan- 

53 

ned monthly rate and their PERT networks reports showed no 

54 

significant schedule slippage. Schedule network reporting 
based on starts rather than completion, xmtimely or inaccur^* 
ate reporting, and the lack of performance of scheduled non- 
critical path work all served to build in cost overruns which 
often went undiscovered until it was too late to take any 
meaningful cost or schedule control action. 

In 1967 the Department of Defense issued a directive 

55 

title "Performance Measurement for Selected Acquisitions". 

56 

The system is summarized as follows 

"Part One of the program requires that contractors 
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use internal planning and control systems that 
meet minimum government criteria. These criteria 
are called the * Cost and Schedule Control Systems 
Criteria' (CSCSC). 

Part Two of the program requires that contractors 
regularly submit Cost PerCormance Reports which 
contain information on the budgeted value of work 
performed to date. (The criteria themselves do 
not require the submission of any reports to the 
government, but specify the reporting capabilities 
which contractors' internal systems must have, and 
the types of data which the systems should be able 
to produce.) The contractor is free to design his 
internal planning and control systems to correspond 
to the manner in which he organizes his work units 
and assigns responsibility for performing work." 

An important concept in the reporting criteria is the 

contract cost status reports must be based strictly on the 

57 

number of Jobs completed to date. 

Five years after the CSCSC had been developed, only 

l6 defense contractors had been certified as complying with 

the criteria. The Navy was singled out by the Senate Armed 

Service Committee as being particularly slow in implementing 
58 

the new system. In 1971 an additional twenty contractors 
complied with the criteria and all three services were act- 
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ively Implementing the program and training personnel in its 
59 

use. 



CURRENT PERIOD RESULTS 

It is certainly too early to make any factual observa* 
tions about the results achieved by the reforms instituted 
since I 969 - 7 O in the ship acquisition process. The Guided 
Missile Frigate program, as Indicated, pioneered many of 
these reforms in the Navy. In addition to those aspects of 
the Guided Missile Frigate acquisition plan already discus- 
sed, a key element of the plan was to select a ”lead’^ and 
a ’‘secondary" contractor early in the design effort. The 
function of the “lead" shipbuilder was to assist in the 
"in-house" design effort and ultimately to build the first 
ship under a "cost plus" type contract. The purpose of this 
was to Introduce producabillty into the design, to promote 
design familiarity and acceptance of performance responsibil- 

60 

ity by the contractor, and to reduce development time. 

The "secondary" shipbuilder was Involved to prevent the 
introduction of producabillty bias by the '*lead“ shipbuilder 
which would result in unfair advantage when bidding on the 
follow ship contracts and to provide a "fallback" position 
in case lead ship contract negotiations turned sour. When the 
first increment follow ships fixed price contract bids were 
received in late summer 1975 j the Navy was dismayed to 
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receive bids from only two contractors - the designated 
’’lead” and ’’secondary” contractors. Moreover, the bid prices 
were well in excess of the ”Design-to«Cost” constraint, • 
reportedly as much as 6o^ over. It can be speculated that 
the lack of participation in the bidding by other shipbuild - 
ers and the high bids submitted were due to one or more of 
the following reasons 

a) shipyard loading by other (primarily merchant 
ship) contracts, 

b) the poor profits and losses experience on previous 
contracts, 

c) a reluctance to accept the required engagement by 
the government in the contractor* s procedures and operations, 

d) fear that the escalation provided for in the con- 
tracts for inflation will be insufficient, as it has been in 
the past, 

and, 

e) the Navy*s cost estimate was far too low. 

Whether or not the current difficulties in the Guided 

Missile Frigate program are a result of past project diffi- 
culties or an indication of the failures of the new reforms 
is indeterminable at the present time. How many Guided 
Missile Frigates (50 were originally planned) will ultimately 
be built, whether or not contractor’s will accept the desired 
increased engagement by the government in ’’their affairs”. 
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?>nd the future of constrained ship design are all open to 
conjecture. The Navy's only other "Deslgn-to-Cost" ship 
class to date, the DG Aegis, was cancelled and replaced by 
the nuclear powered Strike Cruiser (CSGN). Is is an "un- 
constrained design and one not indicative of the 'Design- 
61 

to-Cost' concept.” 

Chapter II will consider the case for a comprehensive 
and structured ship evaluation system. 
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chapter t conclusions and summary 



Three distinguishable oeriods of naval ship acquisition 
policy have occurred since World War II. The periods are 
referred to as the "Conventional", "Concept Formulation/ 
Contract Definition" or "Total Package Procurement", and 
the "Current", 

The aprroaches have differed in the degree of centrali- 
zation of control within the Department of Defense; contrac- 
tor - government relationships; the degree of systems analy- 
sis; whether the design was accomplished by the government, 
private Industry, or by both; and the types and numbers 
of control systems. 

The objectives have remained basically the same: to 
deliver the ships needed for the nation’s defense, when 
they are needed, and at an affordable, promised price. 

The visible results have also been similar - disappointing, 
to say the least. 
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CHAPTER II 



THE CASE FOR A NAVAL SHIP EVALUATION SYSTEM 
INTRODUCTION 

This chapter considers the major factors supporting 
establishment of a ocmprehensive ship evaluation system. 

These factors include the historically short lived acqui- 
sition policies (relative to project duration), over-emphasis 
on short term goal achievement, and the need for a ship 
acquisition life-cycle historical record and data bank. 

It is concluded that successful establishment of a 
comprehensive evaluation system will result in a better 
basis than now exists for formulating high potential acqui- 
sition system improvements and some desirable increase in 
emphasis on longer term goal achievement. A danger in the 
evaluation system will be the tendency to use its results 
for personnel performance appraisal. 

A BETTER BASIS FOR ACQUISITION SYSTEM CHANGES IS NEEDED 

As discussed in Chapter I, defense system acquisition 
policies and strategies have been unstable relative to the 
considerable length of time required to develop and construct 
a new class of ships (often ten to twelve years). Three 



54 



major departures In acquisition policy occured in less time 
than it took to develop and procure the USS NIMITZ, the first 
In a new class of nuclear powered aircraft carriers. The 
primary emphasis in weapons system acquisition policy changed 
from maximum technical performance during the Conventional 
Period, to "cost effectiveness" over the life-cycle during 
Concept Formulation/Contract Definition, to achieving 
acceptable performance for the minimum acquisition cost 
dtiring the Current Period. Although the NIMITZ may partially 
reflect parts of all these policies, the "die was cast" early 
in its development, under the Conventional Period policies. 

The Navy is paying a large price for the adaptability 
and ease of conversion (to minimize life-cycle costs) design- 
ed into the new SPRUANCE class destroyers (DD-963 class) 
under construction by Litton. This was a keystone of the 
Concept Formulation/Contract Definition strategy. The new 
Guided Missile Frigate class, however, is a "tight" design 
with small margins (reservations for future systems weight 
and volume requirements). This is a direct result of the 
acquisition cost and displacement constraints placed on the 
design resulting from the Current Period policies. 

It is easy to forget when judging a ship which policies 
were in effect when it was designed and constructed (and 
also what the acquisition environment was like). Without a 
formalized, consistent evaluation system which collects and 
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assesses data on the actual results of the acquisitions 
throughout the ships’ life-cycles, the Navy will never know 
the degree of successes and failures realized under the 
various acquisition policies and strategies. Without this 
knowledge, changes to the acquisition system are reactionary 
and often little more than "shots in the dark". 

INCREASED EMPHASIS ON LONG TERM GOAL ACHIEVEMENT ^ NEEDED 

Short term goals in ship acquisition include those 
concerned with acquisition cost and delivery schedule. 

Longer term goals include minimizing total life cycle costs 
and providing capability in the design to adapt successfully 
to new and unforseen mission requirements and to counter 
future threats. Both types of goals are Important, depending 
on the requirements of the need. The realities of the 
political/economic environment strongly influence the 
selection of goals and their attainment, as does the need 
requirements. The following few sections will focus on 
the reasons for dominant emphasis on short term goals and 
why longer term goals may be more important. 

SHORT TERM GOAL ATTAINMENT PRESSURES 

There are a number of reasons why short term goals 
achievement receives more emphasis in ship acquisition than 
longer term goals achievement. Short term goals, generally 
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attainable during the acquisition phase, are more easily- 
quantified and measured. The short tenure (relative to the 
development and acquisition time) of key participants increa- 
ses the pressure to optimize short term goals. A typical 
project will have two or three different project managers. 

They are likely to achieve or fail promotion based on the 
results of the acquisition foreseeable during their tenure. 

One Ship Acquisition Project Manager interviewed by the 
author noted that "For a successful project in this building, 
bring it in within cost and on schedule. Unfortunately, the 
fleet will not see it that way." 

By "cost" the project manager meant "acquisition cost", 
and "in this building" refers to the parent organization of 
the project, the Naval Sea Systems Command. The last line 
indicates the project manager's awareness that the operators 
are concerned with longer term goals. A senior Naval Material 
Command (parent organization of the Naval Sea Systems Command) 
executive recently prepared "lessons learned" from two troub- 
led projects. He noted that 

"Our current reward system does not encourage special 
attention to that portion of the program which will 
occur 'down the pike' on a subsequent project mana- 
ger's watch. Our reward system recognizes near term 
achievement. With the increased tour lengths of pro- 
ject managers, this problem has been jpartially alleviated." 
The Concept Formula tion/Contract Definition period policy 
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directive required minimum tour lengths of three years for 
1 

project managers. Recent emphasis has been towards tours of 
four or five years with relief occurring at major milestones. 

The tenures of other key procurement participants are 
normally less than the project’s life. Key officials in the 
Navy and the Department of Defense , military or civilian, 
rarely are in office more than three or four years. Few 
Congressmen, particularly Representatives, can afford to 
emphasize long terra goal achievement at the expense of short 
term, visible results -as s result of their frequent rerlec- 
tion requirements. Additionally, Congress doer not presently 
have the capability to assess the long term results of acqui- 
sition programs, Tlie end result is that ma.ny military and 
public officials whe sponsor and approve the acquisition of 

new defense systems are not around to evaluate the program 
2 

results . 



Economic R ealities 

The move away from minimizing life-cycle costs and tow- 
ards tight control of acquisition costs by specifying acqui- 
sition cost constraints as design parameters was precipitated 
by 

a) the monumental cost overruns of the 1960’s, 

b) diminishing real resources available for new 
acquisitions 
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and 



c) a rapidly aging and diminishing fleet. 

A recent study of the Navy’s acquisition system noted 

3 

that 

” Changing national priorities have reduced 
the percentage of the Federal Budget now 
devoted to defense to the lowest point 
since 1946." 
and 9 

" the U.S. Navy today possesses the 

lowest force level It has Imown since before 

1950 " 



Reporting and Control Systems Effects 

4 

Anthony observes that In business^ 

"Management control systems tend to measure 
current performance, rather than long term 
performance. That Is, they measure the 
profit earned this year, rather than the 
effect on future profits of actions taken 
this year. " 

There Is certainly a directly analagous situation In the 
defense systems acquisition process. The Selected Acquisi- 
tion Reports, submitted quarterly to Congress by major weap- 
ons projects, contain a lengthy analysis of variances from 
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acquieltlon cost estimates. The sections on technical per- 
formance estimate changes is brief and there is no provision 

5 

for life cycle costs in the reports. 

Funding 

New weapons systems are funded annually by Congress. 

The Shipbuilding and Conversion, Navy (SCN) account carries 
a five year obligation authority for the annual appropria- 
tions. This account, until recently, covered new ship devel- 
opment and production. Recently, funding of new ship devel- 
opment and construction of the lead ship of a class has been 
provided by the Research, Development, Testing and Evaluation 
(R,D,T&E) account. This budget category only authoriaes a 
single year obligational authority. Long range planning is 
extremely difficult under such a short funding time horizon. 
Erratic congressional action on the yearly appropriations is 
not uncommon, and this compounds the problem and serves to 
emphasize short term goal achievement. 

THE IMPORTANCE OP LONG TERM GOALS 

The threat of obsolescence in new ships at the time 
they are delivered for service is real. The missions and 
threats they are designed to cope with may change dramatically 
during the long development and acquisition period. In the 
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average ship’s thirty years service life, missions, threats 
and available systems will continue to change as a result 
of rapid strides in technology. Certainly the ultimately 
important test of ship performance is how it performs the 
missions it is called upon to perform, whenever that might 
be. 

The cost to develop and acquire a new ship is typically 
only about one total expenditures over the life 

cycle of the ship. Over emphasis on the short term goal of 
minimum acquisition cost can not help but negatively impact 
the total life cycle cost. Increased automation in a ship 
results in smaller manning requirements, but the automation 
increases the acquisition cost. Larger manning requirements 
however, significantly impact the operating costs both 
through increased ship 6ize (as much as 750 cubic feet per 
man) and the dramatically increasing costs of wages and 
benefits. Another example concerns the long life, low main* 
tenance coatings and materials becoming available which are 
costly to purchase but may favorably impact total life cycle 
cost. Chief of Naval Operations Zumwalt realized the severe 
impact on the increasingly limited Navy resources of the 
high operating and maintenance costs of the aging fleet. He 
embarked on a plan to eliminate ’fapiidly many of the costly 
old ships in order to free funds for new ship construction. 

In conclusion, ship acquisition long term goals (e.g. 
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minimum total life cycle costs and capability to adapt 
rapidly to new threats and missions) are important but the 
balance of pressures on the acquisition process are towards 
short term goal achievement. A consistent and comprehensive 
evaluation system^ operable throughout the ship’s life cycle, 
can help shift this balance by evaluating and making visible 
the degree of attainment of long term goals. 

A COMPLETE HISTORICAL RECORD I£ HEEDED 

The recent effort to improve cost estimating within 
the Department of Defense (see Chapter I) by the use of 
comparative parametric cost estimates has been heunpered by 
the lack of historical return costs of previous weapons 
acquisitions. Engineering cost estimates and actual design 
control are developed along the lines of a work breakdown 
structure in accordance with Military Standard 88l, ’’Work 
Breakdown Structures for Defense Material Items”. 

However, the accrual accounting system established 
July 1, 197^, for the procurement appropriations uses a 
different cost category breakdown which does not correlate 
easily ^with the work breakdown structures used by the pro^ 
Jects. A recent effort to develop a crosb referencing sys- 
tem between the design and engineering cost estimating work 
breakdown structures and the accounting system breakdown 
failed. An additional difficulty in building a return cost 
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historical data base is the previously discussed contractor 
reluctance to divulge actual costs. 

The current increased emphasis on traceability of per- 
formance > cost and schedule estimates was noted in Chapter I. 
Historical records of the project are required to be main- 
tained by the project staff but the records may or may not 
be available to new projects and are of varying formats 
and quality. 

After the acquisition phase, a lot of cost and perform- 
ance data is generated in different accounts and by different 
systems. There are at least a half dozen different cost 
accounts for the Navy which contain parts of the life cycle 
costs of ships. Life cycle costs such as those of mainten- 
ance, training and replenishleent facilities and operations 
are not always identified with the ship classes receiving 
the support. 



PERFORMANCE DATA 

Currently during acquisition, the performance of a ship 
system is being demonstrated more rigorously than ever 
before. Actual hardware demonstrations are replacing paper 
estimates earlier in the process in many cases. At the end 
of the acquisition phase, complete ship trials are conducted 
by the Board of Inspection and Survey to assess the perform- 
ance of the ships. 
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After acquisition, there are numerous sources of inform- 
ation about actual ship performance. In recent years, actual 
combat performance has been available. In peacetime, ship's 
undergo a constant series of inspections of various types. 
They participate in fleet training exercises, undergo ship 
’’refresher training", and compete with other ships in their 
operating groups for coveted "Efficiency" awards which are 
based on the outcome of a series of standard exercises. The 
Board of Inspection and Survey and the Propulsion Examining 
Boards conduct periodic comprehensive inspections of material 
and operating readiness. Information about the reliability, 
maintainabiliy and performance of equipments is generated 
from the "3M" system of scheduled preventive ipiialntenaaee and 
corrective maintenance reporting. Operational ships report 
the status of their capabilities to perform mission require- 
ments through the Joint Chiefs of Staff Force Status and 
Identity Reports. Significant degradations due to equipment 

and system casualties or personnel deficiencies are reported 

9 

using the "Naval Combat Readiness Criteria". 

Prom the above and other sources, pertinent data c 
about the performance of the ships and their costs are 
generated. To the author's knowledge however, there is no 
concerted effort to collect and assimilate all the pertinent 
data and utilize it to evaluate the long term worth of the 
ships. Such a purpose would be served by a central evalua- 
tion system. 



THE EVALUATION SYSTEM AND PERFORMANCE APPRAISAL 



The evaluation system is not being proposed as a control 
system. Control systemsare short horizon oriented and there 
are sufficient and entirely adequate systems (see Chapter I) 
in existence to help manage the acquisition process and eval- 
uate the cost effectiveness of goal achievement during the 
10 

acquisition phase. 

There are several reasons why the evaluation system to 
be outlined in the next chapter should not be used ftr personnel 
performance appraisal. Personnel appraisal demands a degree 
of fairness and objectivity that will not be attainable in a 
system that is designed to evaluate the long term results of 
a process as complex and long lived as a ship acquisition. 

It may be true that the responsibility of management is to 
be fair to subordinates, not necessarily to design a person- 
nel appraisal system that is completely fair and objective 
11 12 
in itself. However, as noted by Christensen, et al, 

"Qy far the most important problem of measure- 
ment is that increased Interest in the measure- 
ment of performance against standards brings 
increased danger that the executive evaluation 
program may encourage performance which detracts 
from rather thaui supports the overall strategy.” 

This tendency results in actions which are directed 
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towards optimizing the performance system results at the 
expense of the projects goals, particularly long term goals. 
The more quantitative and simplistic the evaluation criteria 
the greater is the tendency. A common example which aurises 
in business is the evaluation of profit center managers on 
simple return on investment. The reluctance of an executive 
so evaluated to invest in new, more efficient but higher 
book value production equipment is understandable but is 
of ten detrimental to the long term profit performance of the 
firm. 

13 

Lorange discusses the problem this way 

"There is a danger that the performance tracking 

process will lead to decision-matking behavior 

which violates the overall organizational goal 

congruence requirement. This is partly due to 

the "technical" measurement problems 

These often stem from a desire to capture 

complex and multifaceted phenomena by means 

of a few variables " 

The true function of a measurement system is to get 

an idea of the problems limiting achievement rather than to 
14 

place blame. No single measure or measurement system 
can totally assess the individuals contribution to intermed- 
iate and long term results or to the efforts of others. This 
is particularly true when the time horizon of long term goals 
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l8 extremely long as in the case of ship acquisitions. As 
an avowed purpose of the evaluation system is to increase 
the emphasis on long term goal achievement, any temptation 
to utilize the system for personnel appraisal should be 
resisted. 



CHAPTER II CONCLUSIONS 

Unless improvement in the ship acquisition process is 
to be left to chance, the degree of attainment of project 
goals must be observed and measured. Currently, by far the 
most emphasis is on attainment of short term goals e.g. 
minimizing acquisition cost. The most important goals in 
ship acquisition may be longer term in nature, e.g. minimi- 
zation of total life cycle costs and performance adaptabil- 
ity. 

A consistent and comprehensive evaluation system which 
assessee the degree of success of ship acquisitions should 
be developed. A key function of the system will be to col- 
lect and integrate the various data concerning life cycle 
costs and performance as the data becomes available for the 
different ship classes. The results contained in this 
information will be evaluated against the goals of the pro- 
ject, long and short term. The resulting assessments will 
be used for formulating improvements to the acquisition 
process. The historical data collected should also be 
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of use for budgeting, improving estimating relationships 
and providing “lessons learned" to new acquisitions. 

The evaluation system should not be used for the apprai- 
sal of personnel performance nor is it intended for acquisi- 
tion phase project control. 

Chapter III will consider the development of a naval 
ship evaluation system, present individual sets of goals 
criteria for the three major goal areas, and discuss two 
approaches to the overall assessment of naval ship success. 
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CHAPTER III 



A SHIP SUCCESS EVALUATION SYSTEM 
INTRODUCTION 

In this chapter a framework for a ship evaluation system 
is outlined. In the first part of the chapter, a discussion 
of the desirable traits of a ship evaluation system is presen- 
ted. The prerequisites of the evaluation system include 
establishment of the need, a set of prioritized goals based 
on the need, and a definition of success in naval ship acqui- 
sition. A preferred definition of success is offered. 

The second part of Chapter III includes development of 
sets of goal criteria for each of the three major goal areas. 
This is followed by outlines of a simple manual model and 
a more sophisticated, computerized simulation model for com- 
bining the sets of criteria and arriving at an overall evalua- 
tion of success. The latter model requires greater resources 
but may result in a more ’valid and objective evaluation 
scheme. 

It is concluded that the proposed evaluation system 
is feasible, can be designed so that the criteria are relevant 
and the assessments arrived at, though not completely objec- 
tive, will be more objective and meaningful than any currently 
available. 
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DESIRABLE SYSTEM TRAITS 



The desirable traits of a ship acquisition evaluation 

system are much the same as those required of accounting 

1 

principles - relevancy, objectivity and feasibility. Rele- 
vant in that the measure used is meaningful and useful, 
objective to the extent that the assessment criteria is not 
influenced by personal biases or Judgments and feasible 
meaning that the costs of implementation and the complexity 
of the system are within reason. 

Total objectivity requires purely quantitative analyses. 
This ideal is unobtainable when evaluating a system as com- 
plex as a ship if relevancy and feasibility are to be main- 
tained. The performance of a naval ship is particularly dif- 
ficult to quantify completely. However, as the stated purpose 
of the evaluation system is to determine success as an aid in 
developing improvements in the acquisition process rather than 
to serve as a control system per se, the necessary sacrifice 
of objectivity is not so serious. The goal will be to utili- 
ze single and combinative quantitative criteria as long as it 
makes sense to do so; at the same time recognizing that employ- 
ment of an ’’irrelevant criterion simply because it lends it- 

2 

self to quantification is a poor exchange for alleged ctjectivltyi’ 

EVALUATION SYSTEM PREREQUISITES 

It is assumed ^hst a legitimate and well defined military 

need has been established and a ship has been selected as the 
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best alternative system tc satisfy that need.^ Although the 
proposed evaluation system has been specifically excluded 
from serving as a conventional control system, the basic 
steps required to formulate it are similar: goal specifica- 
tion, establishment of goal attainment criteria, results data 
collection, and evaluation of the degree of goal attainment. 

GOAL SPECIFICATION 

The first requirement is that a set of goals for the 
weapons system acquisition must be clearly specified. These 
goals are derived directly from the need and should reflect 
the planned resources allocation to meet the need. 

Not only must the goals be clearly specified; they must 
be prioritized, or weighted. The reason for this requirement 
is that major project goals (cost, schedule and performance) 
are Interdependent. A schedule change for example, will inev- 
itably result in a cost change, particularly in an inflation- 
ary economy. The generally inverse relationship between acq- 
uisition cost and total life cycle costs was noted in Chapter 
II, Permanent performance deficiencies may result, in a 
remaining gap in the need which must be filled ultimately. 

It is Impossible to optimize simultaneously interdepen- 
dent goals. Some directives and project plans, however, seem 
to require this. For example, a recent Chief of Naval Opera- 
tions directive notes that there had been little documentation 
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available 



of the effort expended to maximize ships * 

mission effectiveness v/hile at the same time minimizing 

ship costs ." (emphasis added) 

For the most part, however, current policy directives 

refer to a need to achieve a proper balance between major 

goals. Pbr example. Secretary of the Navy Instruction 5000.1 

"System Acquisition in the Department of the Navy", specifies 

"Continual tradeoffs shall be performed to optimize 

the balance between Initial acquisition cost, 

estimated life cycle cost, schedule, and operational 

capability." (emphasis added) 

Hie proper balance, i.e. the priority of goals, as well 

as actual selection of the goals, should be tailored to each 

project as the requirements of the needs and the priorities 

of the needs are different. Further, the relative importance 

of the goals should be specified prelimlnery to the search for 

the "best" system to fill the need. As noted in the 1972 

5 

"Report of the Commission on Government Procurement", 

"The need should be separated from any particular 
system, and goals should be defined independently 
of the performance, cost and schedule characteristics 

of any particular system the exploration of 

alternative solutions requires that these things 
be specified. 

* A statement of the problem to be 
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solved or the deficiency in mission capa- 
bility, including those conditions that 
have created the problem or deficiency. 

* goals to be achieved by the acqui- 
sition program and their relative import - 
ance , including level of mission capability, 
program cost and when the capability should 
be available. 

♦The boundary conditions that must be met 
by any system, including constraints on phy- 
sical size, operating conditions, tactics, 
and the talents of the users.” 

(emphasis added) 

Once a specific ship acquisition project is formed in 
response to the need, the prioritized goal set should be 
formally specified in the project's charter. This will 
provide the acquisition group with the specific guidance 
needed to conduct consistent tradeoff analyses. 

DEFINITION OP SUCCESS 

Different ship acquisition participants and obervors 
view success differently. As previously noted, some empha- 
size achievement of one or more short term goals, others 
are interested in only the long term worth of the acquisition 
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Assuming that a prioritized goal set is established 

which accurately reflects the requirements of a legitimate 

need, the only definition of success in ship acquisition 

which makes sense is how well the actual results of the 

acquisition match the goals^ including their relative 

6 

importance. The goal set must be flexible to the point 
that it continues to reflect the requirements of a changing 
need. 



CRITERIA CONSIDERATIONS 

The selection of appropriate standards by which to 
measure actual results against is not always clearcut. The 
choice is influenced by the goal in question, the phase of 
the acquisition, the observability and measurability of the 
data, and whether the assessment concerns actual versus 
planned results or one ship class against another. 

Estimates for the major goal areas are required in the 
Operational Requirement Document which is prepared prelimi- 
nary to Conceptual Design. These estimates are changed and 
refined as the design progresses. The estimates become 
thresholds when written into a Secretary of Defense approved 
Decision Coordinating Paper after the first Defense Systems 
Acquisition Review Council presentation (refer to Figure 1-2 
in Chapter I). 
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Most Important to meaningful evaluation Is that changes 
to estimates are fully substantiated as to cause and are 
completely documented. Traceability of estimates permit the 
evaluation results arrived at by using one standard to be 
adjusted for a prior or later standard as desired. 

No one type of criteria Is feasible for the evaluation 
of all the different aspects of a ship acquisition. The 
question of quantified versus qualitative criteria was addres 
sed previously. Performance measures for complex systems are 
generally probabilistic at best, usually In the form of plots 
The necessary combination of such criteria to arrive at 
higher level systems and functions evaluation results In an 
Increasing need for Judgment. 

Measurement of results In the cost area can be mostly 
quantitative and In the form of variance analysis (meaning 
comparison of actual results to planned results rather than 
the statistical definition). The major questions to be an- 
swered are what are the appropriate costs to evaluate and 
what are the appropriate baselines to use. Still, judgment 
of the economic environment In which the acquisition took 
place Is necessary In order to determine controllable and 
uncontrollable effects on costs. 

The assessment of success In the schedule area deals 
primarily with how closely the ships are delivered to the 
schedule determined from the need. Quantitative variance 



77 




If f ITT 



analysis is useful here also. The choice of what is the 
appropriate “required schedule'* may not be dlearcut. 

After a short discussion of data requirements for the 
evaluation system, individual sets of criteria for each of 
the major goal areas will be developed in detail. Then, 
two different approaches to the combination of those criteria, 
so that an overall evaluation of success can be arrived at, 
will be developed. 



DATA REQUIREMENTS 

The evaluation system should have the data available 
that is needed to describe accurately the acquisition and 
permit in-depth assessment of what its results were and why. 
What must be avoided is the collection of data which is dupli- 
cative, irrelevant or more costly to collect than its value. 

It is difficult to imagine that new data systems will be 
required to implement the evaluation system due to the numer- 
ous data sources discussed previously. The new function pro- 
posed is primarily that of collecting and assimilating data. 

Close adherence to DOD Directive 5000. 19 > “Policies for 
the Management and Control of DOD Information Requirements", 
will help ensure data requirements are economical and concise. 
This directive directs that new information requirements be 
screened for suitability, and that they be designed to meet 
only essential needs. 
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SHIP PERFORMANCE EVALUATION 



The performance of the ship system is Key when consider- 

7 

ing the relative worth of ships. An acquisition may be 
delivered on schedule and meet its cost target, but if it 
fails to perform the missions required of it, in no sense 
can it be considered successful. Conversely, in the opinion 
of at least one Ship Acquisition Project Manager interviewed 
by the author, costs and schedule overruns do not remain 
visible for long after delivery of the ships and thus the 
performance of the units is the only goal area which counts 
in the long run. The weakness of this viewpoint lies in the 
insidious effect acquisitions which are out of control can 
have on other acquisition projects, and thus on the overall 
defense posture. Neither short term nor long term goals can 
be ignored. 

SHIP PERFORMANCE DEFINED 

The definition of ship performance must include any 
feature that impacts directly or indirectly on the ability 
of a ship to carry out its assigned mission. This broad 
definition was earlier offered by Graham. Graham includes 
combat ability (first and most Important), mobility, surviva- 
bility, habitability, maintainability, and future capability 
within the scope of the definition. Eckhart points out that 
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any measure of ship effectiveness must include availability - 

9 

"ships at sea versus ships in being", Sanders reiterated 
10 

the point as follows 

"Too many of our ships are only partially 
ready to carry out their missions on a 
continual basis. This places our Fleet 

and Force commanders, into the 

position of assigning tasks on the basis 
of which ship is ready rather than which 
ship has the most built-in capability to 
do the job." 

Depending on how the performance features are defined, 
others may be listed, such as the detectability of the ship. 
Since a change in capability in one feature area inevitably 
Impacts on another, the evaluation of overall ship perform- 
ance must consider many features, some of which are difficult 
to quantify. 

For example, a new requirement to Increase the size of 
the shlp*s crew will Increase, unless compensated for by re- 
ductions in other performance features, the displacement 
and volume of the ship (possibly as much as 5 tons and 750 
cubic feet per man). The increased displacement and size 
will slow the ship down and reduce its fuel endurance. To 
reclaim this loss in mobility performance will require 
increased horsepower in the propulsion plant. This in turn 
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will further decrease the endurance and additional fuel 



tankage will have to be added. The Increased size and weight 
of the propulsion plant an6 fuel tankage Increases the dis- 
placement and volume of the ship, etc. It goes without 
saying that each necessary Increase in the ship will add 
to the acquisition and other life cycle costs. 

Why it is necessary to include all performance features 

when considering ship performance is made clearer to the 

design layman when an analysis of their Impact is made in 

terms of "pure" performance, such as the number of guns, 

11 

as Graham demonstrated. Pbr example, it was found that the 
increased space built into our ship machinery spaces (increa- 
sed maintainability) designed since World War II is equiva- 
lent in impact to two 5*' light weight guns on a representa- 
tive destroyer. Thus the question, simply put, is "Is the 
•tradeoff* of larger specific machinery box with armament, 
representing a compromise of maintainability and hab- 
itability with combat ability, a good one?" The answer to 
this question rests in how well the overall results match ;*: 
the project goals set, l.e., the degree of success. 

A FRAMEWORK FOR PERFORMANCE EVALUATION 

Objective evaluation of complex system performance 
achievement has eluded systems analysts and controllers. As 

8l 
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Baumgartner observed in 1963 j 

"The main question to be resolved at this point 
in time is how to measure performance and how 
to plfioi performance. Effective answers will 
be developed Just as surely as the final 
performance objectives of a project are 
achieved by effort and direction rather 
than by accident." 

It appears that Baumgartner was overly optimistic as no 
completely objective and comprehensive measurement systems 
have been developed, to this writer’s knowledge, that can 
account for different inputs, acquisition environments, and 
output form and purpose. 

However, the development during the past three years of 
the Top Level Requirements (TLR) and Top Level Specifications 
(TTjS) process provides the basic framework needed for a con- 
sistent and comprehensive (if not completely objective) ship 
performance evaluation system. The key, as will be discussed 
later, will be to extend this new requirements setting and 
performance measure specification process beyond the acquisi- 
tion phase. 

The following sections will discuss in detail the Top 
Level Requirements/Top Level Specifications process and the 
contents of the documents. A third key element of the per- 
formance evaluation system to be developed is a system of 
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consistent procedures for calculating the performance meas- 
ures in the Top Level Specifications. An initial effort 
in this direction exists and will be discussed. The existing 
and planned standardized procedures (termed Performance Anal- 
ysis Data Sheets) are presented, as are the Top Level Speci- 
fications performance measures, in a composite table of 
performance features (from the Top Level Requirements) and 
criteria. 

The Top Level Requirements/Top Level Specifications Process 

As noted in Chapter I, a draft Top Level Requirements, 

based on the results of Conceptual Design tradeoff studies 

and cost effectiveness analyses, is issued at the beginning 

13 

of Preliminary Design. The performance requirements in 

excess of minimum essential requirements are then traded off 

against cost and schedule requirements. 

The Top Level Specifications breaks down the performance 

statements in the Top Level Requirements into basic func- 
14 

tions. An operational capability to "search, detect, clas- 
sify and track air targets", for example, is divided into the 

» 

four basic functions listed. A performance measure is used 
for each of the four. The function "search/detect" is logi- 
cally measured by range. The Top Level Requirements may 
specify a minimum essential value or an acceptable range of 
range values. Ihe Top Level Specification, however, will 
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state the predicted range performance . 

A further example is that the basic operational capabil- 
ities in mobility include "speed, endurance, seakeeping qual- 
ities, repair capabilities, etc." For an aircraft carrier 
escort, the Top Level Requirements may specify a minimum es- 
sential speed in a given sea condition but the Top Level Spec- 
ifications will answer with predicted speed performance and 

16 

speed degradation in given sea conditions. 

As the Top Level Specifications "presents the achievable 

ship within the cost constraint, it may well be that some 

performance measures in (the) Top Level Requirements cannot 
17 

be met ..." A Change in the cost constraint, required per- 
formance or design practices must then be negotiated and docu- 
mented in the Top Level Requirements and Top Level ^)«±fkjationa 
At the end of Preliminary Design, "final" Top Level Re- 
quirements and Top Level Specifications (containing design 
specification and performance measures) are issued. Subse- 
quently, all changes to either document are serialized and 
fully documented. (See Figure 1-2.) 

18 

Top Level Requirements Content 
The major sections of the Top Level Requirements include 
Operational Capabilities, Planned Use and Cost Constraints. 

The document is a translation of the military need into speci- 
fic requirements. A brief description of it contents follows. 
A. Operational Capabilities 

1. Warfare Areas . This is a description of the 
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"combat tasks and functions the ship is Intended to perform 
in appropriate mission areas". These mission areas, detailed 
in the Chief of Naval Operations Instruction 3501 .2, "Naval 
Combat Readiness Criteria", include surface warfare, anti-air 
warfare, anti-submarine warfare, etc. llie "warfare areas" 
correspond directly to Graham's "combat ability" performance 
feature. The section includes how the ship relies on and 
complements the capabilities of other types of units in the 
face of particular threats. Requirements related to the de- 
tection of the ship by different sensor types and requirements 
related to the survival of the ship, e.g. "hardness" to shock 
and air blast, are also described in this section. 

Mobility Area. This includes the "operational cap- 
abilities required in the movement and transit of the ship". 
Listed in performance terms are speed, endurance, seakeeping, 
maneuvering, replenishment constraints and other operational 
constraints. Draft and beam restriction ( to allow harbor 
and canal passage) as well as damage control and engineering 
casualty control requirements are described. 

?' C onmand rmd Control Ar^a . This area describes the 
provisions for planning and controlling operations of the ship 
and associated forces pursuant to the assigned missions. The 
extent of the operational capabilities and interfaces with 
other systems (by digital computer on large modern combatants) 
is detailed. 

Support Operations Area. These are the "opera- 
tional capabilities required gf the ship to provide aid and 
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support to other units." 

N on-Coirbat Operations Area . These are the opera- 
tional capabilities expected which are not directly related 
to combat." Included are search and rescue, etc. 

B, Planned Use 

Environment . This area includes a description of 
the ship's expected operating environment as well as its 
allowable effect on the environment. The standard environ- 
mental conditions against which ship and subsystem perform- 
ance are to be predicted as well as pollution related matters 
are detailed in this section. 

Operational Availability and Ship Utilization . 
This section "describes the ship availability and readiness 
to get underway for specific ship readiness conditions in 
normsl and altered conditions." 

3. Maintenance . Overhaul , and Supply Support Con - 
cepts . This section is the intended maintenance and support 
philosophy for the ship. Minor and major repair frequency 
and duration and the level of on board maintenance capability 
and repair parts availability are specified. 

4. Manning . The "manning philosophy, together with 
operational Items which affect manning, such as degree of 
system automation, watch requirements, and experience levels 
and such other personnel related matters which constrain ship 
manning" are detailed here. Habitability and any special 
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safety requirements are included. 

5. Wartime Use , liiis section "describes the anti- 
cipated operational profile and maintenance cycles during 
times of hostilities when the ship is subject to participa- 
tion in combat". Subtopics include speed and time profile, 
maintenance and overhaul cycle, and operating and logistic 
support. 

Peacetime Use . The same information as that 
specified in the above section is presented, but for during 
peacetime. 

7. Other . In this area are planned use items 
which do not clearly fall under the previous six topics. 
Applicable exceptions to existing regulations and directives, 
as well as pertinent directives which are not normally invok- 
ed, are spelled out. 

19 

Top Level Specifications 

The Top Level Specifications, the design answer to the 
required operational capabilities and planned use assumptions 
of the Top Level Requirements, presents measures of ship 
performance and overall ship attributes utilizing three dif- 
ferent classification systems. Ihese systems are measures of 
overall ship attributes achieved, measures of ship system 
functional performance achieved (traces directly to the Top 
Level Requirements Operational Capabilities) and measures of 
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performance for ship subsystems. 

Ihe ship system functional performance measures included 
In the Top Level Specifications are generally the most mean- 
ingful ones to the ship operator. This serves the purpose of 
the evaluation system well as a primary interest is in the 
ship's functional capabilities at the mission performance 
level. The performance in any one mission area, e.g. anti- 
submarine warfare, may then be assessed by combining evalua- 
tions of the relevant functions, e.g. search, detect, clas- 
sify, track and engage, of the anti-submarine warfare con- 
tributing systems. 

In some cases, traceability exists between the Top Level 
Requirements and Top Level Specifications performance meas- 
ures through common attribute names, e.g. speed, seakeeping, 
stability, vulnerability, availability and noise. There also 
exists direct traceability between the Top Level Requirements 
Habitability Standards and the Top Level Specifications Habi- 
tability Summary, the Top Level Requirements Supply Stipport 
Concepts and the Top Level Specifications Supportability, 
etc . 



Performance Analysis Data Sheets 

The Naval Ship Engineering Center, design arm of the 
Naval Sea Systems Command, is currently developing a system 
to provide "the analytical procedures required to calculate 
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the performance measures in the Top Level Specifications for 
20 

a ship design. The purpose of these "Performance Analysis 
Data Sheets" is to "set forth a single method for consistent 
application which will precisely define the Top Level Speci- 
fications performance measures and make visible all assump- 
tions, margins, loss allowances, safety factors, etc., used 
21 

in its calculation". Performance Analysis Data Sheets 
standardized procedures and criteria are not being developed 
where it is considered that adequate standardized guidance 
already exists. 



Performance Measures Set 

By combining the performance requirements and con- 
straints in the Top Level Requirements, the performance pre- 
dictions and measures in the Top Level Specifications, and 
by utilizing the existing and planned standardized criteria 
calculations procedures specified in the Performance Analysis 
Data Sheets, a set of criteria for evaluating a ship’s per- 
formance throughout its life cycle can be developed. Meas- 
urement of "success" in the attainment of performance goals 
can then be accomplished by utilizing primarily existing 
and planned systems. As noted previously, the major change 
required will be to collect actual results data and assess 
the performance against the requirements throughout the life 
cycle rather than just during the acquisition phase. 
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The table in Appendix A contains the proposed set of 
performance features evaluation criteria developed as discus- 
sed above. Included in the table are 

a) the name and a description of the performance 
measures and ship attributes considered useful to the evalua- 
tion system, generally derived from the Top Level Specifica- 
tions, 

b) the Top Level Requirements performance objectives 
and planned use areas which are directly measured by the 
performance measure (indicated by a “T" for direct traceabil- 
ity) or are at least partially evaluated by the performance 
measure (indicated by an ”1*' for influence), 

and, 

c) the standardized procedures for calculating the 
performance measures which have been developed, or are plan- 
ned for development. Note that only the title or a very 
brief description of the Performance Analysis Data Sheets 

or other standardized procedure is listed. Where a procedure 
is considered inadequate by the author for the purposes of 
the evaluation system, or is not known to be planned for 
development, a recommendation is included. Such recommenda- 
tions are enclosed in parenthesis. 

The next two sections will develop criteria of costs and 
schedule goals. Then the composite evaluation of success 
will be addressed. 
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COST EVALUATION 



Acquisition cost is the most visible and controver- 
sial factor in naval ship acquisition. This situation re- 
sults from the monumental cost overruns of the past two 
decades and the huge amounts required to purchase new major 
weapons systems. A single new nuclear powered aircraft 
carrier costs in excess of one billion dollars to acquire. 

Never-the-less, a case was made in the previous chapter 
that the total life cycle cost is the more appropriate cost 
target to minimize. However, selection of the appropriate 
cost goals and their relative importaunce for each project is 
left to the goal setting process. The purpose of the eval- 
uation system is to measure how closely the results match 
the original or current aet of prioritized goals. 

ACQUISITION PHASE COST CRITERIA 

There is no scarcity of cost information during the 
acquisition phase. There is a problem, as previously noted, 
of obtaining contractor return costs. However, as the de- 
gree of engagement with the contractors increases and more 
of them meet the Cost and Schedule Control Systems Criteria 
(see Chapter I), useful return costs information should be- 
come more available and be of greater validity. 

The Selected Acquisition Reports (SARS) required by 
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Congress concentrate on acquisition costs and contain 
detailed acquisition cost variance analyses. Acquisition 
cost baselines for the evaluation system could be chosen 
from the cost estimates reported in the SARS, the Design- 
to-Cost constraints, or the estimates in the Decision 
Coordinating Papers. These estimates should all be in 
basic agreement. 

The Naval Sea Systems Command's "Design-to-Cost Guide 

23 

for new ships" stipulates that major programs will use 
SAR specified guidelines (variance analyses) for explaining 
cost growth. The same directive requires that 

"a historical record of cost constraints, 
cost estimates, actual costs as th^r become 
available, and cost Influencing factors 
will be maintained in order to provide the 
basis for explaining any cost increases." 

The Design- to-Cost constraints are formulated in terms 
of "unit sail-away costs". This is defined as 

" the estimated unit cost of construc- 

tion of follow-on ships as delivered including 
government furnished equipment, and the instal- 
lation and adaption of computer programs but 
excluding non-recurring costs and outfitting 
cost, in constant dollars of a given fiscal 
year, based on a given number of ships in the 
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series and a given delivery schedule.” 

Program acquisition cost variances, as reported in the 

25 

quarterly SARS, are classified as resulting from 

a) quantity change 

b) engineering change 

c) support item change 

d) schedule change 

e) unpredictable change (strikes, acts of God) 

f) economic change 

g) estimating change (due to errors or revisions 
in estimating methodology and relationships) 

h) contract performance incentives 

1) contract cost overruns (\inderruns) 

Ihese reports therefore contain an adequate selection of 
acquisition cost criteria and analyses. 

POST ACQUISITION PHASE COST CRITERIA 

As previously noted. Department of Defense Directive 

5000.1. "Acquisition of Major Defense Systems" requires 

that "cost parameters shall be established which consider 

the cost of acquisition and ownership " (emphasis added). 

The Office of the Secretary of Defense Cost Analysis Improve- 
26 

ment Group charter specifies that the group is responsible 
for providing a review and evaluation of independent and 
program cost estimates submitted at the Defense Systems 
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Acquisition Review Council presentations. The charter fur- 
ther stipulates that 

'^These cost reviews shall consider all elements 
of system costs, including procurement, opera - 
tions and support as appropriate." (emphasis added) 

The "Department of the Navy Cost Analysis Program" was 

27 

announced in March, 1975. The establishing directive re- 
quires that costs be considered 

" for all phases of the weapon system, in- 

cluding the conceptual, definition, acquisition 
and operational." 

The program requires the Chief of Naval Operations to 

a) "Ensure that realistic cost estimates are 
provided for the planning, programming, and budgeting of 
systems and equipment acquisition.", 

b) 'bevelop and maintain a cost data base for initial 
and follow on cost estimating, cost review and validation, 
and budgeting.", 

c) "Provide guidance and develop costing methodology 
to ensure that the total costs of acquisition and ownership 
of weapon systems and force units are available to decision 
makers. " 

and, 

d) "Maintain force costing models to facilitate 
consideration of alternative force structures and marginal 
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changes." (emphasis added) 

Integration of the various sources and categories of 
life cycle costs is required for application of the evalua- 
tion system. The above plans and requirements, if carried 
through, should result in that integration. 

Most of the data will be quantitative, and thus the 
criteria can be quantitative. However, some judgments must 
be interjected as to what indirect costs will be charged 
to a particular ship class. These overhead charges include 
the cost of maintaining and operating support facilities 
such as shipyards, repair ships, replenishment ships, sup- 
ply depots, etc. Also, qualitative analysis of the effects 
of the resource constraints, and the political, economic 
and procurement environments is necessary, as previously 
noted. 

Table III-l provides a summary of the suggested base- 
line estimates for the principal life cycle costs categories. 
Actual costs incurred should be available initially from, 
the Selected Acquisition Reports submitted quarterly by 
the projects, and later from the Chief of Naval Operations 
maintained force costing models and data bank. Schedule 
goal criteria will be addressed in the next section. 



95 



COST CRITERIA SUMMARY 
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TABLE III-l 



include replenishment 




97 



TABLE III-l 



SCHEDULE EVALUATION 



A major conslderatloi% along with low unit acquisition 

cost for the Guided Missile Frigate program was to make 

contract award for production of the lead ship in less than 

28 

two years after the requirement was levied in 1971. Two 
and one half to three years is usually required for this 
process. The design management plan of having the Naval 
Ship Engineering Center immediately start Contract Design 
upon completion of Preliminary Design and soon afterwards 
bring in the lead ship and secondary contractors to aid in 
the design was developed primarily due to the schedule 
urgency. 

The early ballistic missile programs were counted by 

most observors as successes rather than failures even though 

they incurred large cost overruns and there were deficien*- 

cles in their initial operational performance. They were 

"successful” because the goals during the "missile gap” era 

put primary emphasis on the early development of some sort 

29 

of deterrent capability. A lUlND Corporation study of the 
ICBM programs character! zed them as am example "where an im- 
mediate demonstration force is needed whether or not it 
30 

works. ” 

Again, the point to be made is that success should be 
measured against the requirements of the need as expressed 
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in a prioritized goal set. It is entirely possible that, at 
least initially, rapid introduction of the system into the 
operating forces will be the dominant requirement. 

As is the case for acquisitjbosn costs, the delivery sched- 
ule of a new ship class is monitored In the Decision Coordi- 
nating Paper/Defense Systems Acquisition Review Council 
and the Selected Acquisition Report processes. Delivery 
schedules are often unstable as a result of one or more of 
the below reasons, 

a) overly optimistic estimates of development and 
construction time, 

b) cost increases which force reductions and delays 
in awarding production contracts, 

and, 

c) yearly fluctuations in the number of ships author- 
ized by Congress. 

The actual delivery schedule is known well before actual 
life cycle costs and mission performance results. The recom- 
•..jnended criteria is a simple variance analysis, as required 
by the Selected Acquisition Reports, using the baseline 
estimate as contained in the Decision Coordinating Paper. 
Schedule variances resulting from the causes cited above 
will require qualitative analyses of their effects. 

In the remainder of this chapter, the combination of 
criteria into an overall success evaluation system is acdressed. 
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OVERALL EVALUATION 



Having selected approptiate individual criteria for 
features of each of the three major goals, it remains to 
assess the combined results of the measure. One of the 
prerequisites specified was that clearly defined, priori- 
tized goals would be stated for a project as derived from 
the established need. This can be done by simply ranking 
the goals in order of priority or by assigning weighting 
factors to the goals, 

STich a specification of goal priorities can be viewed 
as the utility function, or criterion, to be optimized in 
the new ship class. As the goal weighting is determined 
from the requirements of the need, it may be necessary to 
change the respective weightings as the ship class progres- 
ses through its life cycle. A key element of success of a 
given project will be flexibility enough in the design to 
adapt to new mission requirements and resource constraints. 
Comprehensive documentation and tracking of the goal speci- 
fications will permit the evaluators to judge! Intel l,egfently 
the degree of success relative to the current or past goal 
sets, or utility functions. 
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ipiEPENDENT EVALUATION OF MAJOR GOALS 

A strong case has repeatedly been made against evalua- 
ting one or the other major goals singally. ^l^o vpei^’formance 
level is worth any cost, and likewise, inadequate perform- 
ance to accomplish the mission requirements is no bargain. 

The requirements of the need and the relative priority of 
the need to other defense needs must be considered. Only 
by evaluating total results against the prioritized goal 
set can actual success be judged. 

This is not to say, however, that the comprehensive 
data collected on ship acquisitions should not be utilized 
for comparing results in any particular area or for improv- 
ing single design procedures and historically based estima- 
ting relationships. Such uses should be some of the major 
benefits of the evaluation system. 

OVERALL EVALUATION MODELS 

Two approaches to integrating the criteria into an 
overall evaluation model will now be discussed. The approa- 
ches are basically the same, but differ considerably in their 
sophistication and degree of automation. 
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A Manual Evaluation Model 



Existing and already planned systems have been utili- 
zed to a great extent in formulating individual goal feature 
criteria and for data sources for the evaluation system. It 
is, in fact, the emergence of these new systems which have 
made comprehensive evaluation throughout a ship's life cycle 
possible . 

Tn order to combine the interdependent criteria and 
goals, some sort of model must be utilized. The first ap- 
proach to constructing and using an evaluation model contin- 
ues the low additional resource requirement theme. In ad- 
dition to low resource requirements, a simple approach 
which relies heavily on the judgment of the evaluators 
recognizee the following, 

a) the extreme complexities of the system being eval- 
uated, 

b) the intricate interdependencies of cost, perform- 
suice and schedule goals, 

c) the difficulties of finding relevant quantified 
measures for every goal area, 

d) the uniqueness of every project which wiH require 
a different evaluation model in each case, 

and, 

e) the problem of reducing every measure to a common 
dimension to permit straightforward combination. 
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These problems raise serious doubts about the validity and 
usefulness of a computerized model which would basically only 
require data input and then, with little evaluator mahipula- 
tion, turn out an overall assessment of success. 

A simpler, manual scheme will probably produce as good 
or better results at lower cost. The qualitative Judgments 
necessary to make the results meaningful negate the value 
of a computer aid. Additionally, the assessments will be 
more acceptable to acquisition process participants, due 
to the relatively simpler approach used to arrive at them. 

So go the arguments. 

The Basic Scheme 

In each goal area, the myriad of data Inputs would be 
collected and assessed using the individual criteria settled 
upon at the Inception of the project. The evaluation criteria 
should have assigned to them distributed weights indicating 
the relative Importance of each subsystem or subarea being 
measured. These criteria and the weighting of them should 
be developed by the acquisition project and approved through 
the Decision Coordinating Paper/Defense Systems Acquisition 
Review Council process. The criteria presented in the pre- 
vious sections should be useful. 

This is not a completely new requirement for the pro- 
jects. Department of Defense Instruction 7041 . 3 , ’'Economic 
Analysis and Program Evaluation for Resource Management", 
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requires that, 

"Criteria by which effectiveness can be evaluated 
are to be clearly specified at the inception of 
a program/project." 

Development of the criteria set and assignment of relative 
importance weightings to the criteria by the project should 
promote acceptance of the evaluation results. 

A number of schemes, including weighted point assign- 
ments, weighted variances or simply rankings, could then be 
devised which would serve to integrate the weighted criteria 
within each goal area. This is the same type process used 
by many projects to evaluate contractor proposals submitted 
in response to bid requests. Finally, by applying the 
weights contained in the utility function (prioritized goal 
set) to the results in each of the goal areas, an evaluation 
of the overall success of the project is possible. 

A great deal of judgment will be required in assigning 
the weights and applying the criteria. The evaluation 
results will have to be qualified in detail. The subjective 
nature of the assessment, though a "number" is arrived at, 
should never be lost d.ght of. The final resulting evalua- 
tion, never-the-less , should be far superior to the frag- 
mented, biased judgments emanating from various acquisition 
observors and participants. 
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A Brief Example 



Table III-2 presents a sampling of example quantita- 
tive analyses for a fictitious ship class. The methods of 
calculating the goal features variances are arbitrary and 
other methods may be preferable. 

After completion of the individual analyses, the next 
step is to assess the quantitative variances together with 
all relevant subjective information. Tlie variances may be 
adjusted (reasoning should be carefully documented) or sim- 
ply accompanied by modifying qualitative analyses. 

The aggregation of variances to higher levels of goal 
features can then be accomplished by using pre-determined 
weightings. For example, consider the Costs goals. Table 
III-3 indicates how the evaluators might calculate weighted 
variances which indicate the degrees of attainment of the 
Costs goals. The sum of the weighted variances is then 
an Indication of the degree of attainment of the prioritized 
Costs goals. 

Table III-4 is an example of how a final figure which 
indicates the degree of success of the acquisition (i.e. how 
well the results match the prioritized goals set) can be 
arrived at. In the example, a negative {undesirable) overall 
variance of .02 from the prioritized goal set is calculated. 

This example exercise is only an indication of the 
simple quantitative techniques which might be of use to the 

105 



GOAL PEATORK CALCULATION PROCEDURE BASELINE CURRENT VARIANCE 




106 



three combat situations. 
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Additional Procurement Table III-2 
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Table III-2 



Table III-3 

EXAMPLE COSTS GOAL ATTAINMENT CALCULATION 



Cost Category 


Weighting 


Variance 


Weighted 

Variance 


Program Acquisition 
Cost 


.3 


(•17) 


(•05) 


logistic Support/ 
Additional Procurement .1 


.06 


.01 


Costs 

q. Operating/Suppcrt 
Costs 




.02 


.01 


4. Conversion Costs 


.2 


(•07) 


(•01) 



Overall Costs Weighted Variance (*04) 
Table TTI-4 

EXAMPLE OVERALL GOAL ATTAINMENT CALCITLATION 



Goal 


Weighting 


Varl ance 


Weighted Variance 


Cost 


.3 


(.04) 


(.01) 


Performance 


.5 


.00 


.00 


Schedule 


.2 


(•03) 


Loii 



Overall Weighted Variance (.02) 



no 



evaluators. The admonitions in the previous section against 
over-emphasizing or misusing the results of such an analysis 
merit constant awareness. 

31 

A Sophisticate d Evaluation Model 

Advocacy of a more sophisticated evaluation model for 
ship acquisition success evaluation centers around the same 
observations of the complexities involved as made in support 
of the manual model. However, there is evidence that non- 
quantifiable goals and goal attainment criteria are often 
largely ignored when evaluation is done on a primarily intui- 
tive basis. Goals criteria which can not be easily quanti- 
fied should be evaluated by knowledgeable experts and assign- 
ed quantified, though subjective, ratings. Truly non-quanti- 
fiable goal criteria will serve little useful purpose as part 
of pertinent criteria used to determine the degree of success 
attained by a ship acquisition project. 

Also, the limitation of human mentality severely limits 
the number of goal interdependencies that can be analysed 
under the previous model. The decreasing costs of time 
sharing and batch process computer systems, the steady 
advancement of new systems analysis techniques, and the 
previously noted continued expansion of defense acquisition 
information and control systems (creation, processing and 
preservation of large amounts of quantitative and qualitative 
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data), all suggest the feasibility of a sophisticated 
computerized simulation model. 

Beginning in the early 1960’s, the Department of Defense 
has made extensive use of systems analysis computer models 
to aid in appraising alternative weapon systems and to 
integrate long term planning with yearly budgeting. This 
proposal amounts to an extension of the usefulness of the 
long range planning models to include follow up evaluation 
of actual results fit to the requirements. 

That the final assessment of success will be done by 
the model, independent of Judgment, is not advocated. As 

32 

Hitch stated in 1967, 

"Systems analysis Is simply a method to get 
before the decision maker the relevant data, 
organized in the way most useful to him. It 
is no substitute for sound and experienced 
military Judgment, and it is but one of the 
many kinds of Information needed by the 
decision maker." 

Model Construction 

The first step in building the model is the same as 
that for the simpler, manual model. The set of prioritized 
goals (the utility function) for the project is specified. 
This is the success reference against which the overall 
results will be assessed. 

Early in the project life, management scientists will 
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conduct a series of interviews with key design, financial 
and management personnel associated with the project. Itie 
purpose of their interviews will be to discover the inter- 
dependencies of the project goals, subsystem and subgoal 
relationships, and the predicted range of effects on project 
outcome from externalities such as the funding schedule and 
amounts, the procurement and economic environments, etc. 

The relationships will be designed into the simulation 
model. The selected criteria will be weighted and entered. 
Performance measures represented by probabilistic plots will 
be modeled by mathematical functions. Constraints which 
describe cost ceilings and minimum performance requirements 
will be used to flag "unacceptable" results. 

The net contribution that subgoals attainment will 
make towards each of the multiple goals must be completely 
structured in the model. This can be accomplished by first 
specifying separate probability distributions for every 
underlying factor and then specifying factor interdependen- 
cies to form joint probability distributions. 

The individual simulations of each of the major goal 

areas can then be jointly simulated by the same techniques 

as above. This approach is along the same lines as the 

prototype intertemporal goal programming model outlined 
33 

by Ijiri. 
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Model Usage 

As data on actual results become available, they will 
be Inputted to compare with the original estimated probabil- 
ity distributions and point estimates. The latter can be 
modified to reflect actual results and an increasingly 
accurate assessment of success obtained. 

It is doubtful that any ships will ever encounter the 
exact operational and combat scenarios set up in the Top 
Level Specifications. However, the scenarios can be modeled 
in the war gaming activities of the Chief of Naval Operations 
organization. Actual ship performance results can be extrap- 
olated into the war gaming data bank as they become available 
and actual scenario performance predicted with increasing 
accuracy. These war gaming results can then be compared 
to the baseline distributions in the evaluation model. 

The computer model results in combination with qualita- 
tive information available to the evaluators should enable 
a relatively straightforward assessment of overall project 
success. By studying the varying degrees of success of 
different projects in conjunction with the projects' differ- 
ent Imposed acquisition policies, acquisition environments 
and management and design techniques, Intellegent formulation 
of high success potential improvements to the acquisition 
system process and structure should be enhanced. 

The construction and implementation of the computer 

models should be done with as much involvement of project 
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personnel as possible. Obviously, special management science 
skills are needed. The operations research/systems analy- 
sis personnel utilized should have project experience where 
possible, and be primarily management oriented. A great 
deal of interaction between acquisition participants and 
the model builders is an absolute must if the model is to 
be valid and considered useful. 

CHAPTER III CONCLUSIONS AND SUMMARY 

A ship evaluation system should be objective, relevant 
and feasible. Total objectivity is prevented by the complex 
interdependencies and multi-variables involved which are 
often difficult to quantify independently, much less when in 
combination. 

Prerequisites to ship success evaluation include a 
legitimate defense need which is best served by a naval 
ship, a prioritized set of cost, schedule and performance 
goals derived from the need requirements, and definition 
of what success is. Success is best determined by how well 
actual naval ship results, throughout the ship life cycle, 
fit the prioritized goal set. 

The final steps in developing the evaluation system are 
criteria selection, evaluation model construction, and 
previsions for necessary data collection. Criteria and 
data requirements can, for the most part, be met by existing 
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and already planned rysteins. 

Two approaches, differing in resource requirements, 
sophistication and degree of automation, have been presented 
for evaluating "overall success". The use of a "manual 
model" for overall evaluation, it is argued, is preferable 
because validity of the results requires a great deal of 
judgment be interjected into the system. 

The computerized simulation program model, augmented 
by the evaluators' Judgment, can account for more of the 
complex interdependencies involved in evaluating ship 
success. However, It requires greater resources to develop 
and use and its results may not be as acceptable to acquisi- 
tion system participants and observers. 

Neither of the models measure up to complete objectivity 
We never supposed they would. The manual model is certainly 
feasible and could probably be implemented in a short time. 
The greatest problem would probably be setting up the col- 
lection of the myriad of life-cycle data in a central loca- 
ton and processing it to usable form. 

The computerized model is less feasible in that it 
would take a significantly greater amount of resources to 
implement and manage. It Is definitely possible over a 
period of a few years, however. The relevancy of either 
model is dependent on the skill of those involved in the 
construction and implementation of the evaluation system. 
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Although the work required and Its complexity may be 
apalling to some, the implementation of either of the two 
approaches could result in great payoffs for future acquisi- 
tion efforts. 



117 



NOTES TO CHAPTER III 



1. Robert, N. Anthony, Management Accounting , (Homewood, Illi- 
nois, Richard D. Irwin, Inc., 1970, Fourth Edition). 

2. C. Roland Christensen, Kenneth R. Andrews, and Joseph 
L. Bower, Business Policy : Text and Cases , (Homewood, 
Illinois, Richard D. Irwin, Inc., 1973> Third Edition). 

3. For requirements for economic analysis of ongoing and 
alternative defense investments, see Department of Defense 
Instruction 7041.3, "Economic Analysis and Program Evalua- 
tion for Resources Management", October l8, 1972. 

4. Chief of Naval Operations Instruction 9010.300, "Top Level 
Requirements and Top Level Specifications for the develop- 
ment of naval ships", January 4, 1974. 

5. Subcommittee on Federal Spending Practices, Efficiency, 
and Open Government cf the Committee on Government Operation^ 
United States Senate, "Extract from Report on the Commission 
on Government Procurement; Part C - Acquisitbn of Major 
Systems", (Washington, U.S. Government Printing Office, 

March, 1975). 

6. Ibid. 

7. Clark Graham, "The Operator and the Engineer", Naval 
Engineers Jou rnal , Vol. 85, No. 3, June, 1973* 

8. Ibid. 

9. Myron Eckhart, Jr., "To Renew a Navy", Naval Engineers 
Journal , Vol. 82, No. 4, August, 1970. 
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10. Frank P. Sanders, "Improved Readiness - Innovation 

and Greater Dedication", a banquet address printed in Naval 
Engineers Journal , Vol. 85, No. 3, June, 1973. 

11. Graham, op cit. 

12. John S. Baumgartner, "P roject Management ", (Homewood, 
Illinois, Richard D. Irwin, Inc., I963). 

13. Chief of Naval Operations Instruction 9010. 300, op cit. 

14. Conmiander, Naval Sea Systems Command, Instruction 906O.I, 
"Top level Specifications (TLS) for new ship design". May 6, 

1975. 

15. Ibid. 

16. Ibid. 

17. Ibid. 

18. Chief of Naval Operations Instruction 90IO.3OO, op cit. 

19. Commander, Naval Sea Systems Command, Instruction 9060.3, 
op cit. 

20. Naval Ships Engineering Center, "Performance Analysis 
Data Sheets (PADS) Library", draft, unpublished. 

21. Ibid. 

22. Secretary of the Navy Instruction 7700. 5B, "Selected 
Acquisition Report (SAR), February 26, 1972. 

23. Draft, unumbered and undated. 

24. Ibid. 

25. Secretary of the Navy Instruction 7700. 3B, op cit. 

26. Department of Defense Directive 5000.4, "OSD Cost 



119 



Analysis Improvement Group", June 13, 1973. 

27. Secretary of the Navy Instruction 7000. 19B, "Department 
of the Navy Cost Analysis Program", March 12, 1975. 

28. John W. Newcomb and Anthony R. Di Trapani , "The Patrol 
Frigate Program - A New Approach to Ship Design and 
Acquisition", Naval Engineers Jou rnal , Vol. 85, No. 4, August 

1973. 

29. Report of the Commission on Government Procurement, 
December, 1972, op cit. 

30. Ibid. 

31. The test in this section and the simulation model out- 
lined is developed directly from E. Eugene Carter, "Portfolio 
Aspects of Strategic Planning", Journal of Business Policy , 
Vol. 2, No. 4, 1972. 

32. C. J. Hitch, Decision Making for Defense , (Berkeley and 
Los Angeles, University of California Press, I967), as ' 
quoted in Carter, op cit. 

33. Yuji IJiri, Management Goals and Accounting for Control , 
(Chicago, Rand McNally and Company, I965) 

as referenced in Carter, op cit. 



RECOMMENDATTOriS 



Tt is recommended that a comprehensive ship life-cycle 
success evaluation system be developed and implemented. 

The framework provided in Chapter III for such a system 
should be roughly adhered to. Particularly important to 
follow are the concepts of utilizing existing systems out- 
puts and insistence on clearly specified and prioritized 
goals at the inception of a pro.ject. 

The standardized procedures for calculating the Top 
Level Specifications performance measures require further 
development and expansion to presently uncovered areas. 

The Performance Analysis Data Sheets provide a usable frame- 
work for this effort. 

The decision on whether to take the '‘manual" model 
or computerized model approach for development of the 
overall success evaluation system should be carefully 
weighed. It would probably be prudent to start' with^ the 
lower resource requirement system and then transition it 
towards the more sophisticated, computerized model. The 
analysis of this decision and a comprehensive trial imple- 
mentation of a ship success evaluation system is recommended 
for a feature thesis effort. 
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KEY TO TABLE 



"TLS" 

"TLR” 



Top Level Specifications 
Top Level Requirements 
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